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ABSTRACT 



This report presents third-year findings from an evaluation 
of the School Choice Scholarships Foundation Program. In 1997, this program 
provided scholarships via a lottery to low-income. New York City children in 
grades 1 through 4 that allowed them to transfer to private schools. The 
evaluation compared scholarship to control students, using test score, 
survey, and school-level data collected in the year 2000. On standardized 
tests, students offered scholarships performed similarly to control students. 
Those who ever attended private schools did not outperform those who never 
attended private schools. Patterns of impact for Hispanic students differed 
markedly from patterns for African American students in regard to test 
scores. Parents reported that schools and classes attended by scholarship 
students were smaller than those attended by public school students. Private 
schools were more orderly than public schools. Parents of children who 
switched to private schools were much more satisfied with their schools than 
parents of public school students. Among students offered scholarships, 53 
percent used them to attend private schools for 3 full years. Parents who 
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declined scholarships generally did so because they could not afford the 
added tuition and expenses. Five appendixes include research data and 
findings. (Contains 26 tables and 38 references.) (SM) 
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EXECUTIVE SUMMARY 



In 1997, the School Choice Scholarships Foundation (SCSF) announced that it would 
provide 1,300 scholarships so that children of low-income families in grades K-4 in the New 
York City public schools could transfer to private schools. Each scholarship, or “voucher,” was 
worth up to $1,400 annually and could be used for up to four years at a religious or a secular 
school. The SCSF received applications from more than 20,000 students from February through 
April 1997. From the pool of applicants, scholarship recipients were selected in a lottery held in 
May 1997. 

This report presents the third-year findings from an evaluation of the SCSF program in 
which students were randomly assigned to a treatment group (scholarship group) or a control 
group. The evaluation findings are particularly relevant to the current national debate about the 
impacts of vouchers on students and parents — especially in that the SCSF program is one of the 
largest of the current voucher programs in terms of enrollment and has yielded results for a 
racially and ethnically diverse population of low-income students. Similar randomized field 
trials of school voucher programs have been conducted in Dayton, Ohio, and in Washington, 
D.C. This summary highlights the key evaluation findings and briefly describes the study. 



KEY FINDINGS ON OUTCOMES 
Impacts on Test Scores After Three Years 

• On standardized tests, students offered a scholarship generally performed at about the 
same level as students in the control group. More specifically, we used the Iowa Test 
of Basic Skills to assess students’ performance in reading and mathematics and foimd 
that, overall, students in both groups performed about the same. Moreover, those who 
ever attended a private school did not perform at higher levels than those who never 
attended a private school. Nor did those who attended a private school for three full 
years perform at higher levels than those who did never attended a private school. 

• The pattern of impacts for Latino students, however, differs markedly from the 
pattern for African American students. We foimd no impact of a scholarship offer or 
of attending a private school on the test scores of Latino students, but we found a 
significant impact on the test scores of African American students. After three years 
the composite test scores (a combination of math and reading) of African American 
students who were offered a scholarship were about 5.5 percentile points higher than 
the composite test scores of African Americans not offered a scholarship. The 
composite test scores of African American students who ever attended a private 
school (for one, two, or three years) was 7.6 points higher than the composite test 
scores of students who had never attended a private school. The composite test 
scores of African American students who attended a private school for three full years 
was 9.2 percentile points higher than the scores of students who had never attended a 
private school. Impacts of a voucher offer do not vary significantly by grade level. 
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• After the first year of the program, the overall impact of a voucher offer on composite 
test scores for African American students was 4.4 percentile points; after two years, 
the impact was 3.2 percentile points; and after three years, it was 5.5 percentile 
points. Changes in the impact of actually attending a private school were larger, 
starting at 5.7 percentile points in year one, falling to 4.4 points in year two, and then 
rising to 9.2 points in year three. The differences between years one and two and 
years one and three were not statistically significant, but the difference between years 
two and three was. 

• There was no change in the impact of being offered a voucher or attending a private 
school on Latino test scores over time. 



Impacts on School Facilities, School Climate, Parents’ Satisfaction with Schools, and 
Parental Communication and Involvement 

• As reported by parents, the schools attended by the scholarship students were smaller 
than the schools attended by public school students (382 students versus 519 students 
in each type of school, respectively). Class size was smaller as well — there were two 
fewer students in the private-school classrooms than in the public-school classrooms 
(26 students versus 28 students). Private schools were less likely than public schools 
to have a cafeteria, a nurse’s office, or special programs for non-English speakers and 
students with learning problems. On the other hand, private schools were more likely 
to have computer laboratories, after-school programs, and tutors for individual 
students. No differences were found in music or art programs, or in programs for 
advanced learners. Nor were there differences found in the availability of child 
counselors, a gymnasium, or a library. 

• Private schools were more orderly than public schools, according to parents. 
Compared with public school parents, private school parents were less likely to report 
that the following were serious problems at their child’s school: students destroying 
property, tardiness, missing classes, fighting, cheating, and racial conflict. For 
example, 64 percent of the parents with a child in public school reported that fighting 
was a serious problem compared with 34 percent of the parents with a child in private 
school. 

• Private-school students reported better learning conditions at their school than did 
public-school students. Sixty-five percent of private-school students said that 
students get along with teachers while only 49 percent of public-school students said 
the same. Private-school students were also more likely to report that students are 
proud to attend their school and that behavior rules are strict. They were also less 
likely to feel put down by teachers or to report a lot of cheating by other students. 

• Students in private schools were asked to complete more homework than students in 
public schools. Sixty-four percent of the parents with a child in private school said 
that their child had more than an hour of homework per day, compared with 41 
percent of the parents with a child in public school. 
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• Parents of students in private schools said that they received more communication 
from their school about their children than did parents in public schools. 

• Parents with a child in private school will be less involved in their child’s education 
than parents with a child in public school. For example, parents of public-school 
students reported that they helped their child with homework an average of 1 1 times a 
month, compared with 9 times a month for private-school parents. 

• Compared with public-school students, private-school students were more likely to 
participate in church youth groups and attend religious services. 

• Parents of children who switched from public to private schools were much more 
satisfied with their schools than parents of children who remained in the public 
schools. For example, when asked to grade their schools, nearly 42 percent of the 
parents with a child in private school gave their school an “A,” compared with just 10 
percent of parents with a child in public school. 



KEY FINDINGS ON PARTICIPATION IN THE SCSF PROGRAM 

• Among those offered a scholarship, 53 percent used it to attend a private school for 
three full years, 9 percent used it for the first two years but not the third, 12 percent 
used it only in the first year, 2 percent used it only in the second year, and 24 percent 
never used it. 

• Parents who declined a scholarship most frequently gave the following reasons for 
doing so: they could not afford the added tuition and expenses not covered by the 
voucher (45 percent), they could not find a school in a convenient location (33 
percent), and their child had special needs (14 percent). 

• There are many similarities and some differences between the parents and students 
who used the scholarship for at least one year and those who did not. Baseline test 
scores were similar for scholarship takers and decliners; scholarship takers and 
decliners, and their parents were equally likely to have lived at their current residence 
for two years; and mothers of takers and decliners were equally likely to have been 
bom in the United States. On the other hand, scholarship decliners were somewhat 
less likely than scholarship takers to have received special education services before 
the baseline testing session; mothers of scholarship takers were more likely to have 
attended college for some amount of time; and the average income of families of 
scholarship takers was $2,400 higher than that of scholarship decliners. 

• Students who attended private school were no more likely than those who remained 
in public school to move from one school to another. Parent reports indicate that 
similar percentages of public and private school students remained in the same school 
throughout the school year. Similarly, the percentage of students who plaimed to 
attend the same school the next year was similar for the two groups. In contrast, 
public school students were more likely to “graduate” from one school level to the 
next, perhaps because private schools are more likely to have grades K-8 in the same 
school. Suspension rates for students in private school were less than those for 
students in public school. 



THE EVALUATION 



The evaluation of the SCSF program in New York City presented a unique opportunity to 
examine the impact of vouchers on students and parents for students switching to private 
schools. New York City has not only a racially and ethnically diverse population but also the 
largest school system in the nation. We computed the effects of vouchers on education outcomes 
by using a randomized experimental design, which allowed us to compare two statistically 
equivalent groups of students and thereby isolate the unique effect of vouchers on the outcomes 
of interests, including student test scores, school climate and facilities, and parents’ involvement 
and communication with schools. 

Mathematica Policy Research, Inc. (MPR) along with researchers at Harvard University and 
the Program on Education Policy and Governance at Harvard have joined together in conducting 
this evaluation, which includes data collection, analysis, and the reporting of annual findings. 
MPR has collected data four times on the same students and families since 1997 (1997, 1998, 
1 999, and 2000). For instance. The Iowa Test of Basic skills was given to students to measure 
their academic achievement in reading and mathematics. In addition, parents and students 
completed surveys so that we could learn more about students’ educational experiences, parents’ 
experiences with the schools, and their school-related plans for the upcoming year. The student 
response rate for each test administration was moderately high — 100, 78, 65, and 67 percent in 
1997, 1998, 1999, and 2000, respectively. The response rates were somewhat higher for the 
parent and student surveys than for the achievement tests. 
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I. INTRODUCTION 



In this report, we present results from the third follow-up data collection for the evaluation 
of the School Choice Scholarships Foundation program. This privately funded voucher program, 
begun in 1996 by the School Choice Scholarships Foundation (SCSF), was designed to allow for 
the collection of high-quality information about student test-score outcomes and for parental and 
student assessments of public and private schools. Taking advantage of the fact that scholarships 
were awarded by lottery, the evaluation was designed as a randomized field trial. 

The evaluation findings reported in this document are based on test-score data and survey 
data collected in spring 2000. The findings build on the data already collected at baseline and at 
each of the first two follow-up periods in 1998 and 1999. The spring 2000 findings are also 
augmented with school-level data for public schools in New York City. Mathematica Policy 
Research, Inc. (MPR) has joined with researchers at Harvard University’s Program on Education 
Policy and Governance to conduct this evaluation, which includes data collection, analysis, and 
reporting of the evaluatioii findings. 

Many researchers, interest groups, political leaders, and policy analysts have debated the 
desirability of continuing and expanding school choice programs (Brandi 1998; Coulson 
forthcoming; Cobb 1992; Bonsteel and Bonilla 1977; Peterson and Hassel 1998; Ascher, 
Fruchter, and Beme 1996; Carnegie Foundation for the Advancement of Teaching 1992; 
Gutmann 1987; Levin 1998; Fuller and Elmore with Orfield 1996; Rasell and Rothstein 1993; 
Cookson 1995). Unfortunately, high-quality information that might inform this debate is limited. 
Although many studies comparing public and private schools have been published, they have 
been criticized for comparing dissimilar populations. And despite statistical adjustments made to 
account for the dissimilarities, it remains unclear whether findings reflect actual differences 
between public and private schools or simply differences between both the students who attend 
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the schools and the families of the two student populations (Coleman, Hoffer, and Kilgore 1982; 
Chubb and Moe 1990; Neal 1996; Goldberger and Cain 1982; Wilms 1985). 

The best way to make sure that two populations are similar on the broadest possible set of 
characteristics is to assign individuals randomly to treatment and control groups. Recently, a 
number of education studies have used random assignment studies to estimate the effects of 
various education interventions. For instance, the Tennessee STAR study, which randomly 
assigned students to small or large classes, found that smaller classes had positive effects on 
students in kindergarten and first grade (Mosteller 1995; Krueger 1999). Another example of the 
random assignment of students to treatment and control groups is the national evaluation of 
Upward Bound, funded by the U.S. Department of Education, which has examined program 
effects on a variety of outcomes, including credits earned in high school subjects (such as 
mathematics and science) and college enrollment (Myers and Schirm 1999). 

Until recently, random assignment has not been used to study the question of school choice. 
After the SCSF evaluation was initiated, two other random assignment evaluations were started 
in Washington, D.C., and Dayton, Ohio (Howell et al. 2000). 

In view of its following characteristics, the SCSF program provided an opportunity to 
conduct a rigorous random assignment study; 

• A lottery randomly allocated scholarships to applicants. 

• An independent evaluation team administered the lottery. 

• Before the lottery, baseline data on student test performance and family background 
characteristics were collected from the students and their families. 

• Follow-up survey and test data were collected aimually for several years after 
students were offered scholarships. 
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In this report, we provide information about the SCSF program three years after students 
started using their school-choice scholarships. We describe the program sponsored by the SCSF; 
the data collection, analysis, and reporting procedures used by the evaluation team; and findings 
from the evaluation for the third year of the program. 

II. PROGRAM DESCRIPTION 

In February 1997, SCSF announced that it would provide 1,300 scholarships in the amount 
of $1,400 annually for at least three years to children from low-income families currently 
attending public schools. The scholarship could be applied toward the cost of attending a private 
school, either religious or secular. After announcing the program, SCSF received initial 
application forms from over 20,000 students between Febmary and late April 1997. 

To be eligible for a scholarship, children had to be entering grades one through five, living 
in New York City, attending a public school at the time of application, and be a member of a 
family with an income that qualified children for the federal School Lunch Program. To 
ascertain eligibility, students and an adult member of their family had to attend verification 
sessions to document family income and the child’s public-school attendance. 

Because many more families than originally projected applied for scholarships, MPR 
randomly selected families for scholarships through a two-stage procedure. First, as families 
applied for scholarships, they were invited to eligibility assessment and data-collection sessions. 
Initially, all families were invited to the eligibility assessment and data-collection sessions. 
However, after it became clear that more families would be attending the sessions than could be 
accommodated, we began randomly selecting applicants for the sessions. After completion of 
the first stage, families that attended the sessions and met the eligibility requirements were then 
randomly selected for the scholarship or control group. To ensure that all families from the 
different sessions had the same chance of being selected for the scholarship group, we adjusted 
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the second-stage selection probabilities to reflect the differential chances of being invited to the 
verification sessions. 

The final lottery was held in mid-May 1997. MPR administered the lottery; SCSF 
announced the winners. Within the parameters established by SCSF, all applicants had an equal 
chance of winmng the lottery. SCSF, in consultation with the evaluation team, decided in 
advance to allocate 85 percent of the scholarships to applicants from public schools whose 
average test scores were lower than the citywide median. Consequently, applicants from these 
schools, who represented about 70 percent of all applicants, were assigned a higher probability of 
winning a scholarship. In the information reported in the tables in this report, we have adjusted 
the results by weighting cases differentially so that they can be generalized to all eligible 
applicants who would have attended the verification sessions had they been invited regardless of 
whether they attended a low-performing school. After the lottery, SCSF assisted families in 
finding private-school placements. 

m. DATA-COLLECTION PROCEDURES AND RESPONSE RATES 

The evaluation procedures used for the third follow-up data collection were’similar to those 
used during the baseline, first, and second follow-up data collections. Below, we describe the 
procedures and present response rates for each round of data collection. 

A. Collection of Baseline Data 

During the eligibility verification sessions, students were asked to take the Iowa Test of 
Basic Skills (TTBS) in reading and mathematics. Students in kindergarten applying for a 
scholarship for first grade were exempted from the test requirement. Parents were asked to fill 
out questionnaires that asked about their satisfaction with the school their child was currently 
attending, their involvement in their child's education, and their demographic characteristics. 
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The eligibility sessions took place during March, April, and early May 1997 on Saturday 
mornings and on vacation days. The sessions were held at private schools, where students could 
take tests in a classroom setting. In most cases, private-school teachers served as proctors under 
the overall supervision of the staff of MPR. 

While the child was taking the test, the adult accompanying the child to the testing session 
responded to the questionnaire in a separate room. This procedure had the advantage of giving 
parents time to complete the questionnaire and the opportunity to ask any questions concerning 
the meaning of particular questions. Parents were informed that their responses would be held in 
strict confidence and used for statistical purposes only. Questionnaires were available in both 
English and Spanish. 

Given the likelihood that a variety of caretakers might accompany children to the sessions, 
the questions were designed to allow any caretaker familiar with the child’s school experiences 
to respond to the questionnaire. Although grandmothers and other relatives and guardians also 
filled out the questionnaire, a child’s parent answered the questions in over 90 percent of the 
cases. For ease of presentation, we refer to opinions expressed as those of parents. 

Because scholarships were allocated by a lottery, there were few differences between 
students offered and students not offered scholarships (see Appendix A). For example, there 
were no statistically significant differences between the two groups in the mother’s level of 
education, family income, number of children living in the honie, or overall reading or 
mathematics test scores of the children in the study.' 

' These findings differ slightly from those initially reported in Peterson et al. (1997). After 
the first follow-up data collection, we revised our weighting procedures to include post- 
stratification adjustments. While some of the baseline comparisons changed, the first follow-up 
impacts estimates remained fairly stable. 



B. Collection of Third-Year Follow-Up Data 

The lottery created two statistically equivalent groups of families: (1) a scholarship group 
with 1,000 families and (2) a control group with 960 families.^ Families in both groups were 
invited to attend sessions in April, May, and June 2000 in which students again took the ITBS in 
reading and mathematics and parents completed surveys that asked a wide range of questions 
about the educational experiences of their oldest child within the age range eligible for a 
scholarship. Students were also asked to complete short questionnaires. 

Testing and questionnaire administration procedures were similar to those followed diuing 
the baseline and the second follow-up sessions. Both the scholarship students and students in the 
control group were tested in locations other than the school they were currently attending. 

Table 1 shows the response rates for the third follow-up survey and test, along with response 
rates for the baseline and second follow-up sxirveys and test administration (see tables section on 
page 41). We present results separately for the treatment and control groups and distin g uish 
between students who did not attend testing sessions and those who completed too few items on 
the reading and mathematics tests to be scored by the test publisher. Riverside Publi shing 
Seventy-two percent of the families selected for the evaluation participated in the sessions held 
in spring 2000. This satisfactory response rate was achieved in part because SCSF conditioned 
the renewal of scholarships on participation in the evaluation; nonscholarship winners selected to 
become members of the control group were compensated for their expenses and told that they 
could automatically reapply for a new lottery if they participated in the follow-up sessions. 

As shown in Table 1, the participation rate was similar for the treatment and control groups: 
71 percent of the families offered scholarships participated in the evaluation compared with 72 

^ Procedures used to construct the two groups and to collect first-year and second-year 
follow-up information are described in Jennifer Hill, Donald B. Rubin, and Neal Thomas (1997). 
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percent of the families in the control group. Sixty-seven percent of the students participating in 
the evaluation attended the testing session, including 69 percent of those offered scholarships 
compared with 65 percent of students in the control group. 

The percentage of completed tests is lower than the family survey response rate because 
some students did not complete enough questions to have their tests scored and some parents 
who did not attend the testing sessions completed the surveys over the telephone. Several 
.reasons may accoimt for incomplete tests. For example, a few students decided for one reason or 
another that they did not want to complete all the items. In a few cases, sessions concluded 
before students had time to complete a section of the test. Finally, some students had never 
taken a standardized before and may have foimd the experience overwhelming; these students 
were excused from the testing session. 

Although the background characteristics of respondents and noiu'espondents in the third- 
year follow-up, as measured in the baseline survey conducted in 1997, resembled one another in 
most respects, they differed significantly along some dimensions (see Appendix A). Among the 
treatment group, respondents were less likely to be white or black and more likely to be Hispanic 
other than Puerto Rican. Mothers of respondents were more likely than mothers of 
noiu'espondents to have been bom outside the United States, more likely to state their religious 
affiliation as Catholic, and less likely to be food stamp or welfare recipients. Respondents had 
average incomes at baseline of around $9,800 compared with $8,800 for nonrespondents. 
Respondents were also less likely to speak English at home. 

Members of the control group who participated in the third-year follow-up were less likely 
than nonrespondents to be black and more likely to be Hispanic other than Puerto Rican. The 
mothers of respondents were more likely not to be working, and they originally reported an 
average income of around $9,000 compared with $10,000 for nonrespondents. Respondent 




7 






families were also more likely to be collecting social security and Medicaid and less likely to 
speak English at home. Finally, families were more likely to report Catholic religious affiliation 
and to note that their child had received help for a disability. 

To adjust for nonresponse in the statistical analyses, we used an analytic model to predict 
noiu-esponse based on a variety of background characteristics. We then applied the predicted 
probability of not responding to adjust the sample weights. Because differences between 
respondents and non-respondents were small, the addition of these weights had only minor 
effects on estimates of impacts. Appendix B provides a more detailed discussion of the 
procedure. 

IV. DATA ANALYSIS AND REPORTING 

In third-year data were usefiil for answering three questions about the effects of school 
choice: 

1 . What was the impact of the offer of an SCSF scholarship on a group of low-income 
scholarship applicants as measured by test scores and as perceived by the applicants 
themselves? 

2. What was the impact of ever attending a private school, whether for one, two or three 

years? 

3. What was the impact of attending a private school for three full years? Are any 

impacts observed in the first and second years sustained? Do impacts increase with 
the number of years in the program? Or do impacts dissipate? 

The first question asks: what happens when school vouchers are offered to families? What 
are the impacts on the population of low-income families interested in taking advantage of 
school choice? The first question is similar to a question often asked in medical research: What 
will happen if a particular pill is marketed? How will the health of potential users be altered 
whether or not all patients use the pill as prescribed? 
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The second and third questions seek to know what happens when children from low-income 
families switch from a public to a private school. The question provides information concerning 
the consequences of actually attending a private school, not just the effects of an offer to do so. 
In medical research, the parallel questions are: What are the consequences of taking a pill for 
one to three years? What are the consequences of taking a pill for a full three years? 

In addressing all three questions, we took advantage of the fact that a lottery was used to 
award scholarships. That is, we were able to compare two groups of students who were similar, 
on average, in almost all respects except that the members of the control group were not offered 
a scholarship. As a result, differences in outcomes can be attributed to the effects of the 
scholarship offer and not to measured or unmeasured differences between students in the two 
groups. 

The analytic techniques needed to answer each question differ in important ways. The first 
question can be answered straightforwardly by comparing the responses of those who were 
offered a scholarship with the responses of the control group. To compute the impact of a 
scholarship offer on children’s test scores, we estimated statistical models that accounted for the 
following: whether a student was offered a scholarship, baseline reading and mathematics test 
scores, and variables that define the randomization process (that is, blocking or stratification 
variables). Baseline characteristics were included to adjust for chance differences between the 
characteristics of treatment- and control-group members and to increase the precision of the 
estimated impacts. We used a similar approach to compute the impacts of the program as 
measured by the parent and student survey responses. In analytic models predicting parent and 
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student responses, we did not, however, include baseline data other than a treatment indicator 
and variables used to define the randomization process. 

To compute the effects of attending private school on students’ test scores, we estimated a 
statistical model that accounted for baseline test scores, variables used to define the 
randomization process, and whether students attended private or public school. The test score 
analyses provide estimates of the impact of ever switching to private school and the impact of 
attending private school for three years. The impacts of switching to private school were 
estimated by using an instrumental variable (fV) estimator; the instrument was the treatment 
indicator, which was determined by the random assignment process. 

Use of the fV estimator to estimate the impact of ever attending a private school requires us 
to assume that the offer had no impact on the year-three test scores for students who were 
selected for the scholarship and never used it. In other words, we estimate impacts for all those 
who attended private school, whether for one, two or three years. If benefits increase with the 
amount of time a student attended private school, the estimate provides a lower-boxmd estimate 

^ Randomization increases the chances that treatment and control groups will be similar and, 
when the samples are large, that impact estimates will therefore be unbiased. Although including 
baseline test-score data in the survey analysis could have increased precision, it would have 
resulted in loss of data because kindergarten students were not tested at baseline. The test-score 
results focus on students in grades 1-4 at baseline, while the results from the analyses of parent 
and student satisfaction generalize to all who applied and were eligible for a scholarship. 

^ Since all eligible children within a. family could receive a scholarship, some families had 
two or more children in the evaluation. The presence of multiple children from the same famil y 
produces clustering effects. When clustering is present and analyses are conducted under the 
assumption of simple random sampling — that is, that all observations are independent — 
researchers may underestimate the standard error of the estimated impact, overestimate test 
statistics, and conclude inappropriately that a difference between the treatment group and the 
control group is statistically sigmficant. To approxunate the true standard error more precisely, 
we estimated the standard errors for the impact estimates by using the bootstrap method (Stine 
1990; Fox and Long 1982). This method provides a direct estimate of the variability in the 
treatment impact without the need to make an assumption about the independence of the 
observations in the sample. 





of the impact of switching to private school. Use of the IV estimator to estimate the impact of 
attending private school for three years requires us to make an alternative assumption. In this 
estimation, we assume the offer of a scholarship had no average effect, either positive or 
negative, on those who did not use the scholarship for the full three years — the 16 percent of the 
treatment group participating in the third year of the evaluation who did not ever use the 
scholarship or the 6 percent who used the scholarship for just one year, or the 9 percent who used 
it for two years. In other words, the differences in test scores of the treatment group, as 
compared to the control group, is assumed to be produced solely by those remaining in private 
school for three years — and that never attending a private school or switching back and forth 
between public and private schools had, on average, neither a positive or negative effect. If 
switching had positive effects on student test scores, these results over-estimate three-year 
impacts. If this kind of switching had negative effects on student test scores, these results under- 
estimate three-year impacts. 

To compute program impacts on the parent and student outcomes measured with survey 
items, we used a similar approach; however, we did not include the baseline test scores to predict 
parent and student responses. Because parent and student surveys asked for information 
pertaining to the student’s current school, the survey analyses focus on the impact of attending 
private school in year three.* Appendix C provides a detailed description of the model 
estimation procedures. 

Most of the findings presented in this report are statistically similar to the findings presented 
in the first- and second-year reports. To avoid repeating this fact, we make a point of 

* It is important to note the distinction between the survey analysis and test-score analysis. 
The survey analysis focuses on the impact of being in private school in year three while the test- 
score analyses examine the impact of being in private school for three years. 



emphasizing when third-year results differ from first- and second- year results. If we do not point 
out differences in third-year results, then those results are statistically indistinguishable from 
those of earlier years. Appendix C describes the procedures used to compare the results from 
year one with the results from year two and from year one with year three. 

V, A CONTEXT FOR INTERPRETING PROGRAM IMPACTS 

A. Response Bias 

Program impacts based on parental and student surveys should be interpreted in light of 
potential response bias. That is, parents and students may exaggerate their responses to some 
items, such as satisfaction with their schools, time spent on homework, or educational 
expectations. No special weight should be placed on the actual frequency with which any 
particular type of event is said to take place. For example, one should not take too seriously the 
claim by children in third through fifth grades that they spend, on average, approximately one 
hom- and 20 minutes a day on their homework. 

B. Generalization of Findings 

The results of the SCSF program evaluation cannot be generalized to a large-scale voucher 
program that would involve all children in New York City or other central cities because only a 
small number of low-income students in New York City public schools were offered 
scholarships. In addition, these students constitute only a small proportion of the students 
attending New York City private schools. A much larger program serving students from a 
variety of income levels could conceivably have a very different impact. In a larger, better 
funded program, more students might remain in private schools, altering the impact of a voucher 
offer. And more students might take advantage of the opportunity, changing the composition of 
students who would be utilizing the opportunity afforded them. 



Nevertheless, the results of the SCSF program evaluation may say something about the 
likely impact of a small-scale, publicly funded voucher program serving low-income families. 
Earlier research indicates that the family background characteristics of those eligible to receive 
an SCSF scholarship and actual scholarship applicants differ only modestly (Peterson et al. 
1997). 

C. SCSF Program Participation and Members of the Control Group Attending Private 

Schools 

Not all those offered a scholarship attended private school. Before the 1997-1998 school 
year, SCSF offered scholarships to 1,374 children. Administrative records show that, by the end 
of the third year, about 78 percent of these children had ever used a scholarship: 53 percent of 
the children had used a scholarship for three full years, 12 percent used one for two years, and 13 
percent used one for only one year (see Figure 1). Most families who decided not to use a 
scholarship based their decision on fmancial reasons, recognizing that the $1,400 scholarship 
does not cover the full cost of tuition. (Later in this report, we provide a more detailed analysis 
of the reasons given by parents for their school selections and for leaving the program). 

If all children randomly offered a scholarship had attended private school and none of those 
in the control group had done so, then the impact of the offer would be identical to the impact of 
attending private school. However, when not all families offered a scholarship make use of it, 
the estimated impact of the offer is reduced proportionately. Moreover, as shown in Figure 1, we 
found that about 12 percent of the control group attended private school for at least one year: 4 
percent attended for three years, 3 percent attended for two years, and 5 percent attended for one 
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year. Control group members attending private schools also reduced the estimated impact of 
private schools.® 

If all children randomly assigned to the scholarship group had attended private school and 
all children in the control group had attended public school, the “treatment differential” would 
have been 100 percentage points. The private-school attendance patterns of the treatment and 
control groups suggest that the treatment differential is 66 percent for those who ever attended 
private school and 49 percent for those who attended private school for three full years.’ 

D. Participants Versus Nonparticipants 

A frequently expressed concern about school vouchers is that only more advantaged families 
will take advantage of the voucher opportunity. To examine this question we examined those 
who used the scholarship for at least one year (ever takers) compared with those who never used 
the scholarship (never takers). 

Table 2 shows no significant differences in the initial test scores between those who were 
offered and used a scholarship and those who did not use it. We present the scores in terms of 
national percentile rankings (NPR). NPR scores show a student’s relative position or rank 
relative to other students who are in the same grade and who were tested at the- same time of 
year. The initial reading scores collected during the 1997 baseline testing session averaged 23 

® Figure 1 uses administrative data provided by SCSF and pertains to all students in the 
study sample. Among those who participated in the third year evaluation 16 percent of the 
treatment group did not ever use the scholarship, 6 percent used the it for just one year, 9 percent 
used it for two years, and 68 percent used it for three years. 

’ We computed these treatment differentials (49 and 66 points) by using two approaches. 
First, we compared the percent of treatments with three years of exposure to private school (53 
percent) with the percent of the control group that reported attending private school for three 
years (4 percent). Second, we compared the percent of the treatment group that attended private 
school for one or more years (78 percent) with the percent of the control group with the same 
pattern of private school attendance (12 percent). 



NPRs for both the treatment and control groups. In mathematics, the difference — only one NPR 
point — was not a statistically significant. Those who took advantage of the scholarship were 
also just as likely as never takers to have lived at their current residence for two or more years. 

There were, however, several differences between the two groups as related to economic 
well-being. For example, mothers of students ever using the scholarship were more likely to 
have more than a high school education. In particular, 54 percent of mothers of children using 
the scholarship had more than a high school education compared with 45 percent of mothers of 
children not using the scholarship. Economically, those who used the scholarship were in better 
circumstances than those who did not use it. The reported family income of scholarship users 
was about $2,400 higher than that of never takers. Mothers of ever takers were also more likely 
to be working full time, less likely to be on welfare, and less likely to be looking for work. 

Racial/ethnic differences were also apparent. African Americans were more likely to have 
used the scholarship. Whites were less likely to use a scholarship. African Americans made up 
49 percent of those who used the scholarship and 38 percent of those who did not. Whites 
represented 3 percent of those who used a scholarship and 14 percent of those who did not. 
■There is no statistical difference between the percentage of Latinos who did or did not use the 
scholarship. However, 80 percent of those who used the scholarship reported that English was 
the main language spoken in their household compared with only 72 percent of those who did 
not use the scholarship. 

Those who did not use the scholarship were also more likely than those who used it to report 
(at baseline) that their children were receiving special education services related to a disability or 
learning problem (15 versus 10 percent). 

Those who declined the scholarship in year three were asked why they turned down the 
opportunity. As shown in Table 3, they cited three reasons most frequently: they could not 



afford the tuition and expenses (45 percent), they could not find a school in a convenient location 
(33 percent), and their child had special needs (14 percent). About equal proportions of families 
gave the following reasons for declining the offer: the school of their choice did not have 
enough space for their child (7 percent), the quality of the private school was not acceptable (5 
percent), they moved away from the area in which the private school attended by their child was 
located (5 percent), the child was suspended or expelled (5 percent), the child was not asked to 
return next year (5 percent), the private school wanted to hold the child back a grade (4 percent), 
and the child did not pass the private school’s admission test (5 percent). Thirteen percent of the 
families reported “other” reasons for declining the offer. 

The fact that cost was cited most frequently is not surprising given that the $1,400 voucher 
does not cover full tuition and expenses at private schools. The median tuition, according to 
parents of children attending private schools, was $2,000, and the median additional expenses for 
uniforms, school activities, books, supplies, and related items was $500. Therefore, most 
families who accepted an SCSF scholarship needed to find approximately $1,100 per child in 
supplemental funds.^ 

We explored the cost issue by inquiring of parents how they paid the tuition and additional 
expenses if the scholarship did not cover the full cost. We asked parents to list more than one 
source of revenue, if appropriate. Parents most frequently said that family income was the main 
source of funds (80 percent of scholarship users). Twenty-six percent said that relatives and 



* Despite suspicions that families with more children would be less likely to be able to raise 
the supplemental funds and therefore would be more likely to decline the scholarship, a logit 
analysis revealed that no relationship exists between family size and the probability of declining 
a scholarship when other factors were taken into account. 
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fiiends helped out, and 5 percent said that their child had received a separate scholarship directly 
from the school. 

E. Selecting a School 

Critics and proponents of school choice often debate the importance of educational 
considerations in families’ selection of a school. Critics argue that low-income families are more 
concerned about location, sports programs, or religious instmction than they are about academic 
quality per se. For example, the Carnegie Foundation for the Advancement of Teaching said that 
“when parents do select another school, academic concerns often are not central to the decision” 
(Carnegie Foundation for the Advancement of Teaching 1992). Similarly, an American 
Federation of Teachers report on the Cleveland voucher program said that parents sought 
scholarships not because of “‘failing’ public schools” but “for religious reasons or because they 
already had a child attending the same school” (Miuphy, Nelson, and Rosenberg 1997). 
However, some researchers have found that low-income parents, like other parents, place the 
highest priority on the educational quality of the school (Schneider, Teske, Marschall 2000). 

From a list of considerations, parents were asked to select the three most important 
considerations that they had in mind when selecting a school. Parents who accepted the offer of 
a scholarship most frequently mentioned academic quality (listed by nearly 63 percent of the 
parents; see Table 4); their other considerations included school discipline (46 percent of 
parents), religious instruction (34 percent), and teacher quality (32 percent). Almost 30 percent 
listed safety, and 22 percent listed what is taught in class. Almost 20 percent mentioned class 
size and a convenient location. Considerations mentioned by only a small fraction of the parents 
included school facilities, the sports program, and the school attended by the child's friends. 
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In sum, educational considerations seemed predominant, questions of social order 
(discipline and safety) and religious instruction followed, and the facilities and sports program 
were the least important. 

F. Obtaining a Placement in the School of Choice 

Compared with 61 percent of the control group, over 80 percent of parents who received a 
scholarship offer reported success in finding a school that met their needs (see Table 5). Parents 
were asked to identify why they thought that their choice went imfiilfilled (parents could list 
more than one reason). The offer of a scholarship reduced from 31 to 11 percent the proportion 
of parents who said they could not afford their preferred school (see Table 5). Although parents 
most often mentioned the cost factor, families awarded a scholarship gave other reasons for not 
attending a school of their choice, including, in order of frequency, no reason given by the school 
(8 percent of parents), no space available (4 percent), and transportation problems (4 percent). 

VI. THE IMPACT OF THE SCHOLARSHIP PROGRAM ON SCHOOL EXPERIENCES 
One issue of considerable debate is the type of school experience a child will undergo as a 
result of a voucher intervention. Critics of choice say that public schools have better facilities 
and more elaborate programs capable of serving a diverse population and that choice will lead to 
ethnic and racial segregation (Murphy, Nelson, and Rosenberg 1997). Supporters of choice 
claim that private schools have the necessary facilities for and are more successful in integrating 
all children into a conunon framework. They also assert that private schools are, on average, 
more integrated than public schools (Greene 1998). 

To address these issues, we asked parents about the facilities, programs, ethnic composition, 
and disciplinary climate in public and private schools. As noted, because the surveys asked for 
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information pertaining to the student’s current school, the analysis focuses on the impact of 
attending private school in year three. 

A. School Facilities 

The facilities in central-city public schools are expected to be larger, more expensive, and 
more well equipped than the facilities in central-city private schools low income students attend. 
With a few exceptions, reports from applicant parents in New York City are consistent with the 
conventional wisdom. 

As estimated by parents, the effect of choosing a private school reduced the average size of 
their child’s school by 137 students, or over 25 percent — from an average of 519 students to 382 
students (see Table 6). Private-school parents reported at the end of the third year that then- 
children’s classes were smaller. The effect of using a scholarship was to reduce the size of the 
child’s class by two students. 

Parents also reported that private schools were less likely to have a nurse’s office, a 
cafeteria, and special programs for non— English speakers and students with learning problems. 
The greatest difference was in the availability of programs for non— English-speaking students. 
Forty-nine percent of private-school parents reported that such a program existed in their school 
compared with 89 percent of control-group parents. Not all differences were as large; for 
example, 68 percent of private-school families reported that their private school had a program 
for the learning-disabled compared with 82 percent of the parents in the control group (see 
Table 6). 

In a few instances, parents who switched to private school reported that their private schools 
had more extensive facilities and programs. For example, they were somewhat more likely than 
parents in the control group to say that their school had a computer laboratory, individual tutors. 
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and an after-school program. For other facilities and programs, however, such as gymnasiums 
and arts or advanced learner programs, no differences between the two groups were evident. 

In sum, compared with public schools, classes in private schools and the schools themselves 
are smaller, but public schools offer a wider range of facilities and programs. The larger, more 
complex facilities do not, however, seem to satisfy the control-group parents. On the contrary, 
only 9 percent of the parents in the control group were very satisfied with public school facilities, 
as compared to 29 percent of the parents whose children were attending a private school (see 
Table 6). 

B. Ethnic Composition of School 

Switching from public to private school in New York City placed minority students in 
classrooms with majority students, but it also placed more minority students in classrooms which 
were completely made-up of a single race/ethnic group. When asked about the percentage of 
minority students in their child’s classroom, 40 percent of control- group parents replied that 
everyone in the classroom was of minority background (see Table 7). Only 30 percent of the 
private-school parents gave the same response. However, when parents were asked what portion 
of the student's class was of the same race/ethnic background as the child, those in private school 
were slightly more likely to say that everyone was of the same background — 11 percent of 
private-school parents as compared with 6 percent of the control group. Because the first 
question discusses minority students, which could include students from several racial and ethnic 
backgroimds, while the second focuses on the specific racial and ethnic background of the 
classmates, the responses to these two questions are not as inconsistent as they might initially 
appear. 

Moving to private school increased the number of friends of a different race reported by 
students. When asked to indicate the racial background of their four best fnends, the average 
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increase in inter-racial friendships was no less than 0.5 students. Students in private school were 
also just as likely as the control group to welcome a family of a different race next door and to 
reject the notion that people of other races are bad. 

C. Children with Special Needs 

The debate over school choice has, among other issues, focused on special education. 
Critics of school choice say that private schools cannot or do not serve the needs of those with 
physical and mental disabilities (Murphy, Nelson, and Rosenberg 1997). Defenders of school 
choice often claim that many of those diagnosed as disabled can learn in regular classrooms and 
that special arrangements can be made for others. 

To address this question, parents were asked to indicate the biggest obstacle keeping their 
child from performing better at their school. Among the items they were invited to consider was 
the “lack of facilities and programs needed to address their child's special needs.” Parents with 
children in private school were much less likely than control-group parents to say that their 
school lacked such facilities and programs. Only 7 percent of parents with children in private 
school expressed this concern as compared with 17 percent of control-group parents (see 
Table 8). The difference was statistically significant. 

To explore the question of special needs further, we asked parents about their child’s special 
education needs and the availability of school programs to meet those needs. The number of 
learning-disabled and physically disabled students in the evaluation was small; therefore, the 
findings should be interpreted with caution. 

Eleven percent of private-school parents indicated that their child had learning difficulties; 
another 4 percent said that their child had a physical disability (see Table 9). There was very 
little difference reported in how well the schools attended to learning disabilities (21 versus 26 
percent) or physical disabilities (37 versus 39 percent). 



D. School Climate 



If parent reports are accurate, the scholarship program had a substantial impact on the daily 
life of students at school. Private-school parents were more likely to report that the following 
were not a serious problem at their school: destruction of property by students, tardiness, missed 
classes, fighting, cheating, and racial conflict. For example, 34 percent of the parents who 
switched their children to private schools compared with 64 percent of their coimterparts in the 
control group thought that fighting was a serious problem at their child’s school (see Table 10). 
In addition, 39 percent of the parents in the scholarship group perceived tardiness as a problem 
compared with 61 percent of the control-group parents. Twenty-nine percent of private-school 
parents but 48 percent of parents in the control group said that destruction of property was a 
serious problem at their child’s school. 

Student reports of the climate in their school and classroom are not as sharply differentiated 
as those of parents, but they are still consistent with parental assessments. Table 1 1 shows that 
students in private school were more likely than control-group students to report that students 
“get along with teachers” and less likely to say that “there is a lot of cheating in this school.” 

As reported in the first- and second-year evaluation reports, public and private schools seem 
to use different control mechanisms for maintaining discipline (Peterson, Myers, and Howell 
1998; Myers et al. 2000). Private schools placed greater emphasis on dress and orderliness; 
public schools rely more on sign-in sheets and hall passes. Almost all private schools seem to 
require students to wear a school uniform. No less than 97 percent of the parents reported that 
their private school required uniforms as compared with 49 percent of control-group parents (see 
Table 10). Similarly, 95 percent of private-school parents reported that certain types of clothing 
are forbidden, but only 70 percent of control-group parents reported the same. On the other 
hand, parents reported that public schools more frequently use sign-in sheets for parents and 
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visitors and hall passes for students. Ninety-seven percent of the control group reported that 
parents must sign in when they come to school, as compared to 88 percent of private-school 
parents reported such a regulation. To leave their class, control-group students must obtain a hall 
pass, according to about 88 percent of control-group parents, as compared to 79 percent of 
private-school parents mentioned a similar requirement. 

£. Homework 

After three years, parents who switched their children to private school continue to say that 
their children do more homework. Sixty-three percent of parents with a child in private school 
reported that their child had more than one hour of homework a day, whereas only 37 percent of 
control-group parents reported a comparable volume of homework (see Table 12). Private- 
school parents were also less likely than control-group parents to say that homework was too 
easy (6 versus 23 percent). 

Student assessments of their homework were not as sharply differentiated as those of 
parents, but the differences, although statistically insignificant, were in the same direction. 
Students attending private school estimated that they spent, on average, 50 minutes per typical 
night on homework as compared with 45 minutes reported by control-group members (see Table 
12). In one respect, student reports concerning homework differ significantly between the first 
and the second and third years of the program. After one year, students new to private schools 
were more likely than control-group students to report difficulty in keeping up with then- 
homework (Peterson, Myers, and Howell 1998). After two and three years, the difference was no 
longer apparent (see Table 12 and Myers et al. 2000). Students were adjusting to the homework 
expectations of their new school, or the school was adjusting to the new students and changing 
its expectations of the students. 
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F. School Communications with Parents 



Compared with control-group parents, parents of students in private schools said that they 
received more communication from their school about their child. Although no significant 
differences in regular parent-teacher conferences were reported, the data in Table 13 indicate that 
a higher percentage of parents with a child in private school reported the following: 

• Being more informed about the child’s grades halfway through the grading period 

• Being notified when their child is sent to the principal’s office the first time for 
disruptive behavior 

• Speaking to classes about their job 

• Regular parent-teacher conferences 

• Participating in instruction 

• Receiving notes about their child from the teacher 

• Receiving a newsletter about school events 

• Frequency of parents’ nights 

VII. THE IMPACT OF THE SCHOLARSHIP PROGRAM ON PARENT AND 
STUDENT BEHAVIORS AND ATTITUDES 

A. Religious Practices 

Since 98 percent of the scholarship users attended parochial schools, it is possible that a 
switch from public to private schools affected students’ religious practices. Students in private 
schools were more likely than the control group to report that they both attended religious 
services and participated in church groups. Table 14 shows that over 60 percent of the 
scholarship students reported regular attendance at religious services as compared with 37 
percent of students in the control group. Almost half of the scholarship students said that they 
participated in church groups as compared with 28 percent of the students in the control group. 
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However, religious instruction outside the school was just as frequent for the control group as for 
the scholarship students. 

B. Parental Involvement in a Child's Education 

Supporters of school choice claim that when parents actively choose a school, the family 
becomes more engaged in the child’s education and that, together, schools and parents create a 
more effective educational environment for their children (Brandi 1998). Critics of choice argue 
that any observed differences in parental engagement in private schools are attributable to 
parental characteristics that would predispose the parents to greater involvement. 

The evidence after three years provides little indication that school choice increases family 
engagement in education. In fact, in contrast to the first two years of the program when no 
differences in parental involvement in the schools was apparent for scholarship students versus 
the control group, the findings in year three suggest that control-group parents are slightly more 
engaged in their child’s education (see Table 15). Parents were asked how often they helped 
their child with homework, talked with their child about school, attended school activities, and 
worked on school projects. With the exception of working on school projects, parents who 
switched their children to private school were slightly less involved than their counterparts in the 
control group. Control-group parents spent two days more per month helping their children with 
homework, and one day more per month helping their children with reading and mathematics, 
talking to them about school, and attending a school activity with their child. 

C. Student Adjustment to Choice Schools 

Scholarship students appear to have adjusted well to their new schools. They reported the 
same number of friends at school as did control-group students (see Table 16). And they were 
much less likely than control-group students to say that they often “feel made fun of’ by other 
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students. Only 21 percent of the scholarship students versus 35 percent of the control group 
reported being “made fun of” 

D. Peer Influence, Suspension Rates, and Expectations 

A student's close inends may influence his or her educational performance. Table 16 
reveals that private-school students reported a higher number of friends who received good 
grades than did the students in the control group. Further, private-school students were less 
likely than the control group to have as many friends who used bad language. These results are 
similar to those observed in the second year. In other respects, however, no differences in 
fnendship patterns were observed between scholarship students and the control group. The 
number of close friends who liked school, got in trouble with teachers, or smoked cigarettes did 
not differ significantly between the two groups. Nor did the two groups of students report 
different levels of drug and alcohol usage among their close friends. 

Although year-two findings revealed no difference in suspension rates between the two 
groups, the year-three findings show that students who switched to private school were less 
likely than control-group students to be suspended. While 7 percent of control-group parents 
reported that their child was suspended during the year, only 2 percent of parents with a child in 
private school reported a suspension. The difference is statistically significant. 

We foimd no indication that attending private school increases student aspirations. The 
percentage of students reporting that they intended to graduate from high school or to go on to 
post-secondary education after leaving high school was the same for the scholarship users and 
the control-group students. 
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£. Parent and Student Satisfaction 



Most studies of voucher programs for low-income minority families have foimd that 
families taking advantage of a scholarship are much more satisfied with their child’s schooling 
than are families whose child remains in public schools (Peterson 1998; Schneider et al. 1998). 
Our third-year results confirm these findings. As noted above when discussing how schools 
handle students with special needs, we asked parents to name the biggest obstacle that keeps 
their child firom performing better in school (see Table 8). The private-school parents were much 
less likely than the control group parents to cite any obstacle. In fact, 75 percent of parents who 
switched to private schools named no obstacles while only 17 percent of control-group parents 
said claimed no obstacles. The latter were more likely to list the following as obstacles: teacher 
quality (24 versus 5 percent), lack of discipline (17 versus 1 percent), lack of facilities/programs 
to address their child’s special needs (17 versus 7 percent), fnends (10 versus 4 percent), and 
problems in the neighborhood (2 versus 0 percent). 

When asked to assess their school overall, families give higher marks to private schools. 
Over 40 percent of the scholarship users gave their school an “A” compared with 10 percent of 
the control group (see Table 17). 

We also examined parental satisfaction with specific aspects of school life. On every aspect 
about which parents were questioned, parents with a child in private school were substantially 
more satisfied than control-group parents. The percentage of parents “very satisfied” with a 
private school was significantly higher for all of the following: school location, school safety, 
teaching, parental involvement, class size, school facility, student respect for teachers, teacher 
communication with parents with respect to their child’s progress, the extent to which a child can 
observe religious traditions, overall parental support for the school, discipline, clarity of school 



goals, staff teamwork, teaching, academic quality, the sports program, and what is taught in 
school (see Table 17). 

Differences in student reports of satisfaction were in the same direction but not as great as 
those reported by parents. Students in fourth through seventh grades were asked to give their 
school an overall grade. Students in private school were less likely than those in public school to 
give a “D” or an “F.” 

F. Hawthorne Effects 

It may be hypothesized that the SCSF program, like other iimovations, has a Hawthorne 
effect, namely, that iimovation and change alone enhance the level of parental satisfaction. If so, 
then the scholarship program might be expected to have a weaker impact on parental satisfaction 
after three years than after one or two years. In fact, differences in the percentage of private- 
school parents and control-group parents who voiced satisfaction did change significantly from 
year one to year three on 8 of the 16 aspects of school life about which parents were asked. For 
example, the difference between the private-school parents’ and the control-group parents’ 
satisfaction with class size was 33 percentage points at the end of the first year and 22 points at 
the end of the third year — a statistically significant change. Other statistically significant 
changes in parent satisfaction from the first to the third year were related to school facilities, 
discipline, clarity of school goals, parental support for the school, what is taught in the school, 
teaching values, and teamwork among school staff Despite the drop in satisfaction, parents of 
scholarship students remained much more satisfied than control-group parents with every 
dimension of school life they were asked about (see Table 17). 

In addition, parents were asked to give an overall grade to their child’s school. Based on 
their responses, we found no evidence that the program’s impact had a statistically significant 
decline. The impact of private schools on the probability that a parent would give his or her 



O 

ERIC 



29 



39 



child s school an “A” declined from 39 percentile points in year one to 32 percentile points in 
year three. After three years, parents of scholarship users were still much more likely than 
control-group parents to give their child’s school an “A.” 

In sum, some indication suggests that there may have been a slight Hawthorne effect. 
Overall, however, after three years, private-school parents remained more satisfied with their 
child’s school than did public-school parents. 

G. Suspension Rates and School Changes During the School Year 

It is generally thought that students perform better if they can remain in the same school 
throughout the school year and from one year to the next. In the context of the SCSF evaluation, 
then, the question is: Does school choice destabilize a child’s educational experience? In his 
evaluation of the Milwaukee school choice program, John Witte (1991) said that one of his 
concerns was the high rate of attrition from private schools. And a number of critics of choice 
have raised questions about the readiness of private schools to expel students who do not “fit in” 
(Murphy, Nelson, and Rosenberg 1997). Other studies, however, have found that students from 
low-income families are equally likely to remain in the same school both within the school year 
and from one year to the next regardless of whether the school is public or private (Greene et al. 
1998). In general, the findings from the evaluation of the SCSF program confirm the conclusion 
that school choice does not disrupt the education of low-income students. 

As noted, suspension rates in the third year were lower for the private-school group than for 
the control group. Seven percent of the parents in the control group reported that their child had 
been suspended while only 2 percent in the private-school group reported suspensions. 

In all three years, a very high percentage of all students in the study remained in the same 
school for the entire year (see Table 18), and we found no difference between the two groups in 
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school mobility rates.’ In short, school mobility was very low and virtually identical for both 
scholarship users and similar members of the control group. 

H. Plans for Next Year 

Private-school students were more likely than control-group students to plan to attend the 
same school next year. Almost 80 percent of the families of students attending private school 
compared with about 60 percent of control-group families said that they expect their child to be 
back at the same school (see Table 19). However, 16 percent of the control group compared 
with only 5 percent of private-school parents gave “graduating” as the reason for the change in 
schools. Apparently, many of the students in public schools “graduate” from elementary to 
middle school, actually moving from one location to another, whereas private schools do not 
necessarily recognize the transition as a formal graduation and do not require a move to a 
different location. With consideration of differences in the organization of the school system, 
there seems to be no significant difference in mobility rates from one year to the next for the two 
groups of students. 

While private-school parents were less likely than control-group parents to cite graduation 
as a reason for changing schools, they were more likely to cite school cost as a reason for 
changing schools. Four percent of private-school parents reported expense as a factor, although 
none of the controls reported the same. There is no statistically significant difference in the 



’ The percentages may underestimate the actual rate of school mobility for both scholarship 
students and those in the control group. The families that did not attend questionnaire 
administration sessions probably were more likely to have moved, making it more difficult for 
evaluation staff to locate them. If so, the children in those families that could not be located 
would be more likely to have changed schools. In this regard, it is important to note that the 
response rate was lower for the control group than for scholarship users, suggesting that actual 
differences in school mobility rates may be even higher than reported rates. 
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percentage of parents reporting that their children will attend a different school next year as a 
consequence of quality issues, the school’s inconvenient location, or the family’s relocation. 

VIII. THE IMPACT OF THE SCHOLARSHIP PROGRAM ON TEST 
PERFORMANCE 

This third-year evaluation of the SCSF program in New York City provided an opportunity 
to estimate longer-term impacts of school choice on student test scores. Earlier reports describe 
impacts on test scores after one year and two years of program participation (Peterson, Myers, 
and Howell 1998; Myers et al. 2000). This section addresses two general questions about the 
impact of educational vouchers: 

• What is the impact of offering private-school vouchers on students’ academic 
performance? 

• What is the impact of private-school attendance on students ’ academic performance? 

We report on the impacts of the award of a scholarship and the impacts of attending private 
school on student performance on the Iowa Test of Basic Skills in reading and mathematics. We 
report results for each test separately and for the two combined.'® The impact of a scholarship 
offer is reported as the effect on student national percentile rankings (NPR), which may vary 
between 0 and 100. Nationally, the median NPR score on the Iowa Test of Basic Skills is 50. 
For all students, we report the average impact of the offer of a scholarship on the students’ 
combined test-score performance as well as separate estimates on reading and mathematics 



We combined the test scores to form a composite by takmg the average of the reading and 
mathematics scores for each student. Students who had a zero NPR score on either the reading 
or miathematics tests were retained in the analysis. 
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scores (see Tables 20 through 22)." The tables also show average impacts for African American 
and Latino students; the two groups compnsed more than 90 percent of the sample, and there 
were not enough students in other ethnic groups to perform a separate analysis.'^ Besides 
reporting results for all students and for students by race/ethnic group, we report results 
separately for students in grades four, five, six, and seven in year three and for two groups of 
combined grade levels (grades four and five and grades six and seven). As mentioned, we 
estimate the impact of offering a scholarship, the impact of ever switching to private school, and 
the impact of attending private schools for three years. 

A. Impact of a Voucher Offer on Year-Three Test Scores 

The impact of an offer is affected by two factors: the proportion of students who use the 
offer to attend private school and the size of the impact of attendance at a private school among 
those who make use of the offer. As a consequence, one should not interpret the results as 
showing the impact of attending private school; instead, the results show the average impact on 
test scores that we would expect if a policy with parameters and context similar to those foimd in 
the SCSF experiment were implemented. 



" Students who were in kindergarten at the time of the baseline data collection are not 
included in the test score analyses because we did not test such children before offering 
scholarships. 

About 43 percent of those students identified as Latino are Puerto Ricans, about 40 
percent are from the Dominican Republic, and the remaining 1 7 percent are identified as other. 

When describing a grade for a student, we have classified students according to the grade 
they were expected to be in at the end of the third year of the voucher program. Most but not all 
students were in fact in the designated grade; some were held back a grade while others skipped 
a grade. To facilitate accurate comparison, all students were tested as if they were in the 
expected grade. 
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Results from the SCSF evaluation show that, after three years, the offer of a scholarship had 
no overall impact on student performance on tests (see Tables 20 through 22); that is, students 
offered scholarships had about the same test scores as students in the control group. In separate 
analyses of African American and Latino students, however, we found that a scholarship offer 
had a statistically sigmflcant positive impact on test scores of the former but no significant 
impact on test scores of the latter. 

The combined test scores of African American students who received a scholarship offer 
were 5.5 NPR points higher than the combined scores of those not offered a scholarship (effect 
size = 0.20 of a standard deviation). Scores on the reading test for those offered a scholarship 
were almost 4.0 percentile points higher, and in mathematics, scores were 7.0 points higher 
(effect sizes are 0.14 and 0.26 of a standard deviation, respectively). For the combined test 
scores, the difference between the scholarship group and the control group was statistically 
significant. Differences in reading and mathematics scores were also significant. 

We also examined the impacts of a voucher offer for each of the four grades that 
participated in the evaluation. We estimated these impacts both for all students and for African 
American and Hispanic students separately. When estimating impacts by grade level for the two 
ethmc groups, the number of observations was fairly small, ranging between 194 and 188. When 



Comparison of baseline characteristics of students in the treatment group and the control 
group, after three years, reveals some differences (see Appendix A, Table A-1), often for family 
socioeconomic status. For example, for the overall sample of students, we found that the 
baseline reading test scores were higher for the treatment group than for the control group and 
that the treatment group was more likely to receive Medicaid payments. To check the robustness 
of our impact analyses in light of the potential differences between the treatment and control 
groups, we estimated analytic models with the composite test score as an outcome and with the 
standard set of covariates we use in all of the test-score analyses, which includes baseline test 
scores, and we added family income and mother’s educational attainment as predictors. Impact 
estimates with and without the additional predictors are similar, and we arrive at the same 
conclusions. 
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test scores for different ethnic groups are viewed separately by grade level, they demand a 
measure of caution in interpreting the findings. Estimates for ethnic-group and grade-level 
analyses are based on a fairly small number of observations (ranging between 104 and 188 for 
the grade-level comparison with ethnic groups). With numbers this small, statistical estimates 
are susceptible to random fluctuations (Kane, Staiger, and Geppert 2002). In addition, impacts 
must be fairly large before they can be detected at levels of conventionally employed statistical 
significance. When discussing grade-specific impacts, we focus on the effect on composite 
(combined reading and mathematics) test scores because such scores are based on a larger 
number of test items, thereby reducing the amoimt of random fluctuation. 

When we estimated the impact of a scholarship offer separately by grade and for the grade 
groupings for the overall sample of students, we foimd no impacts on the combined test scores. 
Nor did we find grade-specific impacts on the reading or mathematics scores. 

Neither did impacts on African American test scores vary significantly by grade level. In 
ex amining the impact of a voucher offer on African American students by grade level, we 
observed no statistically significant difference in the impacts among grade levels (see Appendix 
D). The average impact on combined test scores for the younger students was 5.0 percentile 
points and 5.5 points for the older grades (see Table 23). Both impacts are statistically 
significant. Further separation of the grade levels into individual grades showed that the 
scholarship offer had statistically significant impacts on the combined test scores of students in 
grades six and seven (effect size = 0.16 and 0.20, respectively) and positive but statistically 
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insignificant impacts for the students in grades four and five.‘* Among Latinos, the scholarship 
offer had no statistically significant impacts on test scores by grade level. 

B. Impact of Attending a Private School on Year-Three Test Scores 

In contrast to considering the offer of a voucher that may have gone unused, our second 
research question focuses on the impact of attending private school. We present two separate 
estimates. First, we describe the impact of ever attending private school. Second, we discuss the 
impact of attending private school for three full years. 

1. Impact of Ever Attending a Private School 

To estimate the impact of ever switching to private school, we .estimate impacts for all those 
who attended private school, whether for one, two, or three years. If benefits increase with the 
amount of time students were in a private school, this underestimates impacts of attending a 
private school for the full three-year period. 

Among all students ever switching to a private school, which includes students who may 
have attended a private school for one, two, or three years, the switch had no significant effect on 
year-three test scores (see Tables 20 through 22). Nor did the switch have any significant effect 
on the test scores of Latino students. 

For African Americans, however, the results were noticeably different, as they were for the 
scholarship offer. The impact of ever switching to private school on the combined test scores of 
Afncan American students was 7.6 NPR points. This impact shows that if African American 
students who sought to leave the public schools in New York City ever switched to private 
school instead of a remaining in the public schools, we would expect, on average, a difference of 

When computing effect sizes for subgroups (Afncan Americans and Latinos and by 
grade), we used the pooled standard deviation for the complete sample. 
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7.6 points between the private-school and public-school groups (effect size = 0.37 of a standard 
deviation). The reading test scores of those ever switching to private school were, on average, 
5.5 NPR points higher, and mathematics scores were 9.7 points higher. The estimate of the 
impacts of attending private school on each of the test-score outcomes was statistically 
significant. 

When the impact of ever attending private school on A&ican American test scores was 
examined by grade level, we observed no statistically significant difference in the size of the 
impacts (see Appendix D). The impacts for grades four and five combined were 6.9 percentile 
points and 7.3 points for students in grades six and seven. Both estimates were statistically 
significant. Within individual grades, the impacts ranged from 3.6 to 9.7 percentile points. Only 
the impacts for African American students in grades six and seven were statistically significant 
when each grade was considered separately. 

2. Impact of Attending a Private School for Three Years 

The previous analysis of the impact of ever switching to private school estimates the average 
impact for students attending private school, whether for one, two, or three years. Ideally, we 
would like to assess the impact of a full, three-year period of private-school attendance relative 
to three years in public school; however, we can only approximate such an impact in this 
evaluation. To compute the impact of attending private school for three years, we had to assume 
that there was no impact of attending for one or two years on third-year test scores; that is, there 
was no benefit or harm in attending private school for one or two years and then returning to 
public school. The results of attending private school for three years are similar to the impacts of 
ever attending private school even though different assumptions were made (see Tables 20 
through 22): (1) among all students, there were no impacts on test scores; (2) among African 
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Americans, substantial impacts were observed; and (3) among Latino students, there were no 
impacts.'* 

For African American students, there was an impact of 9.2 NPRs on the combined test 
scores (effect size = 0.45 of a standard deviation). For the reading test scores, the impact of 
Staying in private school for three years was 6.7 points, and on the mathematics test, the impact 
was 1 1.8 points (effect size = 0.30 and 0.52 of a standard deviation, respectively). 

When the impact of attending private school for three years on African American student 
test scores was examined by grade level, we observed no statistically significant differences in 
the impact between grade levels (see Appendix D). The impact for students in the younger 
grouping was 8.5 percentile points, and in the older grades the average impact was 9.1 points. 
Both impacts were statistically significant. When impacts were estimated separately by grades, 
they ranged frorn 4.4 to 1 1.7 percentile points. Only the impacts for grades six and seven were 
statistically significant. 



C. Impacts of Vouchers over Three Years*’ 

The ideal voucher experiment for elementary school students would show whether the 
impact of attending private schools increased, remained about the same, or declined as students 
moved from the lower to the upper elementary grades. In such an ideal experiment, all students 



’* The assumption used when computing the impact of ever attending a private school was 
that the voucher offer had no impact on those who never attended a private school. The 
assumption underlying the impact of attending for three years was that the voucher offer had no 
impact on those who attended a private school for less than three years. 

Given that the offer of, or use of a voucher, does not have an impact on Latino test scores 
and does have an impact on test scores for African Americans, we conducted a descriptive 
analysis to explore two hypotheses about why the difference in iiiipacts might exist: one focuses 
on differences in observed characteristics of the schools attended by African American students 
and Latino students, and a second examines whether differences exist for first-generation and 
later-generation Latinos. Appendix E presents the results from these analyses. 
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randomly assigned to the voucher group would attend private schools for the duration of the 
experiment, and those in the control group would attend only public schools. However, the ideal 
cannot be achieved since students and families move back and forth between private and public 
schools. To approximate the ideal experiment and assess whether there are changes in impacts, 
we focused on results that show the impacts after years one, two, and three for those who had 
complied with the experimental protocols at the three points in time. 

As might be expected, given the lack of significant impacts for the sample of all students in 
each of the three years (see Tables 23 through 25), we did not observe a statistically signiftcant 
change in impacts between years one and three or among the intervening years (see Table 26).'* 
For Afncan American students, for whom we did find significant average impacts on the 
combined test scores in each year, we did not observe a statistically significant change between 
years one and two or years one and three; however, we did observe a statistically significant 
change between years two and three (see Table 26).'^ When we examined changes in the three 
follow-up years of testing for Afncan American students, we found no significant change in test 
score impacts between years one and two and a statistically significant change between years two 
and three (Table 26).^® The estimated average impact of private-school attendance on the 



'* In Peterson, Myers, and Howell (1998), we reported small positive and statistically 
significant impacts by using the first year of follow-up scores. That analysis relied on one-tailed 
statistical tests while the current report and last year’s report (see Myers et al. 2000) focused on 
two-tailed tests. The use of two-tailed tests in the first-year report would have resulted in 
statistically insignificant overall impacts. We have moved to two-tailed tests in recent years so 
that consumers of the findings can better assess whether there are no impacts, positive impacts, 
or negative impacts. 

’’ Appendix C describes procedures for testing change in impacts across years. 

Although we have focused the discussion of changes in impacts over time on the impacts 
on the composite test scores, we note that for all students and for African American students we 
found a statistically significant change in impacts between year two and year three on the math 
test scores. For all students, the impact on the math scores was about 1 .9 points in year one, -0.6 
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composite test scores of African American students was 5.7 percentile points after one year, 4.4 
points after two years, and 9.2 points after three years. In none of the three years of testing were 
there statistically signifrcant impacts for Latino students, and there were no changes in 
impacts.^ 

Several randomized studies of the impact of vouchers on families and students who apply 
for them have now been completed. Although these studies tell us much about the effects of 
vouchers on those who use them, we are left with many questions about the role of vouchers as a 
public policy. For example, what is the impact of vouchers on students who choose to remain in 
public schools? Will vouchers prompt schools to compete for students? And will such 
competition result in better schools and, in turn, overall higher academic performance for all 
students? Furthermore, the policy debate would benefit from more information about the impact 
of vouchers of varying amounts and about why test scores for some groups go up while the 
scores for others remain unchanged (for example, African Americans versus Latinos, 
respectively). The answers to these questions lie in the continued execution of high-quality 
randomized experiments. 



(continued) 

points in year two, and 2.6 points in year three. For African American students, the impact was 
7 .0 percentile points in year one, 4. 1 points in year two, and 11.8 points in year three. 

Similar results were foimd for (1) the impact of a scholarship offer, (2) the impact of ever 
switching to private school, and (3) the impact of attending for three years. 

As a check for the robustness of the over-time comparisons, we estimated the year-one 
and year-two results by using both the third follow-up and second follow-up sample weights. A 
similar pattern in the year-one and year-two results were observed; that is, somewhat munerically 
larger impacts in year one than in year two. 
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All percentages based on total sample of all families (students) who participated in the second stage of the lottery. 



TABLE 2 



DEMOGRAPHIC CHARACTERISTICS FOR SCHOLARSHIP TAKERS AND DECLINERS" 

(Percentages) 





Ever 

Takers® 

( 1 ) 


Always 

Decliners® 

(2) 


Difference 

( 3 ) 


Family income^ 


Less than S5,000 


27 


38 


-12 


$5 ,000-$ 10,999 


37 


39 


-2 


$ll,000-$24,999 


33 


21 


12*** 


$25,000-$39,999 


3 


2 


1 


S40,000 or more 


0 


0 


0 


Total 


100 


100 




Average family income 

Family receiving government 
assistance 


10,024 


7,590 


2,434*** 


Welfare 


53 


69 


-16*** 


Social Security 


11 


12 


-1 


Mother's employment status^ 


Full time 


24 


15 


9 


Part time 


16 


15 


1 


Looking for work 


44 


55 


-10** 


Not looking 


14 


15 


-1 


Eton’t know 


1 


1 


1 


Total 


99 


101 




Percent of mothers at current residence for two years or 
less 


20 


24 


-5 


Highest level of education completed by mother*** 


Some high school 


21 


24 


-3 


High school graduate or GED certificate 


24 


29 


-5 


Some college 


43 


31 


12*** 


Graduated from a four-year college 


8 


10 


-2 


More than a four-year college degree 


3 


4 


-1 


Don’t know 


1 


3 


-2 


Total 


100 


101 




Mother's ethnicity*** 


Black 


49 


38 


11” 


White 


3 


14 


-6*** 


Puerto Rican 


17 


22 


-6 


Latino other than Puerto Rican 


25 


24 


0 


Other 


5 


2 


0 


Total 


99 


100 
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TABLE 2 {continued) 





Ever 

Takers 

Q) 


Always 

Decliners 

(2) 


Difference 

(3) 


Mother 's religious affiliation^ 


Baptist 


21 


20 


1 


Other Protestant 


18 


12 


1* 


Catholic 


52 


49 


3 


Other religion 


5 


13 




No religion 


4 


7 


-2 


Total 


100 


99 




Percent of mothers U.S.-born 


56 


56 


0 


Percent of households with English as main language 


80 


72 


7** 


Percent of children receiving any special education 
services related to a disability or learning problem 


10 


15 


-5* 


Baseline test scores (in national percentile rankings) 


Reading 


23 


23 


0 

1 


Mathematics 


17 


16 


(N) 


608-1027 


158-293 




Computer Source — MPR: F AY4020 1 .do. 









*Statistically significant at .10. 

** Statistically significant at .05. 

***Statistically significant at .0 1 . 

Significant at .01 using the chi-square; The chi-square test was used to test for differences in the distributions of categorical 
outcomes between takers and decliners. 

"Takers are defined here as students in the treatment group who ever made use of the scholarship; decliners are students in the 
treatment group offered a scholarship but never using it. 
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TABLES 



REASONS WHY STUDENTS DID NOT TAKE SCHOLARSHIPS 



Percentage 



Why is this child not now using his or her scholarship? 

Could not afford the additional tuition and expenses 45 

Could not find a convenient private school 33 

Has special education needs 14 

Other 13 

Was not given space in private school 7 

Quality of private school not acceptable 5 

Moved away from private school ' 5 

Was suspended or expelled from private school 5 

Was asked not to return to private school 5 

Private school wanted to hold child back a grade 4 

Did not pass private-school admissions test 3 

Computer Source— MPR; sty401 1 la.do. 
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TABLE 4 



PERCENTAGE OF SCHOLARSHIP TAKERS WHO RESPONDED THAT THE FOLLOWING 
WAS ONE OF THE THREE MOST IMPORTANT CONDIDERATIONS IN CHOOSING A SCHOOL 



Considerations Percentage 

Academic quality 53 

Discipline 45 

Religious instruction 34 

Teacher quality 32 

Safety 29 

What is taught in class 22 

Convenient location 1 9 

Class size 1 7 

The school was the only choice available 9 

The child went to a neighborhood public school 5 

School facilities 3 

Child’s fnends I 

Sports program q 

Computer Source— MPRE: sty 401 1 l.do. 
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TABLES 



PERCENTAGE OF FAMILIES WHOSE CHILD ATTENDED PREFERRED SCHOOL 

(Percentages) 





Scholarship Offered 


Control Group 


Scholarship Offer 
Impact 


Attended preferred school 


81 


61 


20 "* 


Reasons for not attending preferred school 


Family could not pay the cost 


11 


31 


-20 


Child did not pass admissions test 


1 


1 


0 


Family not member of affiliated church 


1 


1 


0 


No more space at school 


4 


5 


-1 


Applied too late 


2 


2 


0 


Transportation problems 


4 


3 


1 


Moved away from school 


1 


1 


0 


Schoor location inconvenient 


1 


2 


-1 


Conununication problems 


2 


1 


r 


School not in zone or district 


2 


8 


-6 • 


School did not offer special education 


2 


0 


2 • 


No reason given by school 


8 


5 


3 


Other 


1 


1 


0 


On average, how long does it take this child to 


get from home to school each morning?® 


15 


16 


0 


(N) 


1,392 







Computer Source — MPR; fay40I00.do, fey40I02- fay40I05.do. 



"The coding for the length of time it takes to get to school is as follows: S if under 10 minutes, 15 if 1 1-20 minutes, 25 if 21-30 
minutes, 38 if 3 1-45 minutes, 53 if 46*60 minutes, and 61 if more than one hour. 

'impact is statistically significant at .10 level, two-tailed test. 

** Impact is statistically significant at .05 level, two-tailed test. 

*** Impact is statistically significant at .01 level, two-tailed test. 
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TABLE 6 



SIZE AND QUALITY OF SCHOOL FACILITIES 
(Percentages) 





Scholarship 

Offered® 

(1) 


Control 

Group*’ 

_(2) 


Scholarship 

Offer 

Impact^ 

C3) 


Scholarship 

User^ 

C4) 


Control- 

Group 

Complier® 

(5) 


Switch to 
Private School*^ 
(6) 


Average school size 


409 


493 


-84- 


382 


519 


-137- 


Average class size 


24 


25 


-1- 


26 


28 


-2- 


Percentage very satisfied with 
school facilities 


24 


12 


12... 


29 


9 


20*~ 


Percentage with the following 
resources: 














Special programs for non- 
English speakers 


55 


77 


-22'" 


49 


89 


.. A*** 

-40 


Special programs for learning 
disabled 


70 


78 


-8'" 


68 


82 


-14*** 


Nurse’s office 


83 


94 


-11- 


79 


97 


-18*** 


Child counselor 


78 


81 


-3 


77 


83 


-6 


Library 


89 


92 


-3 


87 


91 


-4 


Cafeteria 


91 


96 


-5'" 


88 


96 


-8*** 


Special programs for advanced 
learners 


54 


58 


-4 


51 


58 


-7 


After-school program 


91 


86 


-5'" 


93 


84 


9*** 


Gymnasium 


91 


91 


0 


90 


89 


1 


Arts program 


79 


81 


-2 


82 


85 


-3 


Computer laboratory 


91 


84 


7'" 


93 


81 


12*** 


Music program 


80 


78 


2 


83 


80 


3 


Individual tutors 


^ 57 


44 


13- 


61 


38 


23*** 


(N) 


91^1,379 






91^1,379 







Computer Source — MPR: fay40I02.do, fey40I03d.do. 



‘Tliose who were offered a scholarship, whether or not they made use of it. 
hrhose who were not offered a scholarship. 

^Estimated impact of being offered a scholarship. 

^ose who were offered a scholarship and identified by SCSF staff as having used their scholarship to attend a private school, 
nrhose in the control group who would have used a scholarship had they been offered one as described in Appendix C. 

^Estimated impact of participation in the program during at least the third year, using a two-stage least squares model, as 
described in Appendix C. 

*lmpaa is statistically significant at . 1 0 level, two-tailed test. 

**lmpact is statistically significant at .05 level, two-tailed test. 

***lmpact is statistically significant at .01 level, two-tailed test. 
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TABLE? 



ETHNIC AND RACIAL ISOLATION IN CLASSROOM 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group- 

Complier 

(5) 


Switch to 
Private School 
(6) 


What percentage of students in child's 
class are minority? 














Less than one-quarter 


12 


12 


0 


11 


11 


0 


One-quarter to one-half 


13 


10 


3 


15 


10 


5 


One-half to three-quarters 


11 


13 


-2 


13 


16 


-3 


Three-quarters but not everyone 


30 


25 


5 


30 


22 


8 


Everyone 


34 


40 


-6* 


30 


40 


-10** 


Total 


100 


100 




99 


99 




What portion of the student 's class is 
of the same race/ethnic background as 
this child? 














Less than one-quarter 


19 


17 


2 


19 


16 


3 


One-quarter to one-half 


20 


18 


2 


23 


19 


4 


One-half to three-quarters 


20 


25 


-5* 


20 


28 


-8* 


Three-quarters but not everyone 


28 


31 


-3 


28 


33 


-5 


Everyone 


12 


9 


3* 


11 


6 


5* 


Total 


99 


100 




101 


101 




(N) 


1,353-1,369 






1,353-1,369 






Student reports (percentages) 














Number of fnends of a different race* 


2.8 


2.5 


0.3** 


2.8 


2.3 


0.5** 


Would not like having a family of a 
different race move next door 


10 


12 


-2 


10 


13 


-3 


In general, experiences with people of 
other races have been bad 


6 


9 


-3* 


4 


8 


-4 


(N) 


1,614-1,640 






1,604-1,629 







Computer Source — MPR: fay40I07.do, sty40I I2.do, sty40I I3.do. 
See notes to Table 6. 



niie index is scored 0 if the child reports no fnends of a different race, 1.5 for one to two fhends, 3.5 for three to four fhends, 5.5 for five 
to six fnends, and 7.5 for seven or niore fnends. 



o 

ERIC 
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TABLES 



OBSTACLES THAT KEEP THE CHILD FROM PERFORMING BETTER IN SCHOOL 

(Percentages) 





Scholarship 
Offered 
(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control 

Group- 

Complier 

(5) 


Switch to Private 
School 
(6) 


Parents report as biggest 
obstacle keeping child from 
performing better 














There are no obstacles 


58 


23 


35’’’ 


75 


17 


58’” 


Teacher quality 


8 


19 


-11’” 


5 


24 


-19’” 


Lack of discipline at the school 


5 


15 


-10’” 


1 


17 


-16’” 


Friends 


7 


11 


-4’” 


4 


10 


-6” 


Problems at home 


1 


2 


-1 


0 


2 


-2 


Problems in the neighborhood 


1 


2 


-1” 


0 


2 


-2” 


Lack of motivation 


11 


13 


-2 


8 


12 


-4 


Lack of facilities/programs 
needed to address child’s special 
needs 


10 


16 


-6’” 


7 


17 


-10’” 


(N) 


1,355 






1,355 







Computer Source — MPR: fay40 1 03c.do. 



See notes to Table 6. 
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TABLE 9 



SPECIAL EDUCATION FACILITIES AND PROGRAMS 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Children with physical disabilities 


3 


0 




4 


-I 


5"‘ 


Children with learning disabilities 


10 


6 


4" 


11 


4 


7" 


(N) 


1,392-1,393 






1,392-1393 








Scholarship 

Offered 


Control 

Group 


Difference 


Scholarship 

User 


Control 
Group in 
Public 
Schools 


Difference 


Percentage that believe school 
doing *\ery well** * at attending to 
these need 














Physical disabilities'^ 


.31 


.44 


-.13 


.37 


.39 


-.02 


Learning disabilities'^ 


.24 


.24 


.00 


.21 


.26 


-.04 


(N) 


42-156 






42-156 







Computer Source — MPR: fay40 1 03c.do, fay40 1 04.do, fay40 1 09.do, Iay40 1 1 9.do. 



See notes to Jable 6. 

^ese figures are calculated as a percent of those parents with disabled or non — English-speaking children, not as a percent of 
the entire population. Because these groups were not created by using random assignment, we report differences in means and 
not impacts. 
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TABLE 10 



PARENT PERCEPTIONS OF SCHOOL CLIMATE 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5) 


Switch to 
Private School 
(6) 


Parents report as serious problem 














Fighting 


44 


62 


-18‘“ 


34 


64 


-30“' 


Tardiness 


46 


59 


-13‘“ 


39 


61 


-22'“ 


Students missing class 


37 


52 


-15‘“ 


30 


55 


-25'” 


Students destroying property 


31 


42 


.11- 


29 


48 


-19'" 


Cheating 


35 


41 


-6" 


29 


40 


-11” 


Racial conflict 


31 


36 


-5‘ 


27 


36 


-9' 


Parents report on school rules 














School uniform 


85 


57 


28‘" 


97 


49 


• 

• 

• 

00 


Certain forms of dress forbidden 


87 


72 


15‘“ 


95 


70 


25*” 


Visitors must sign in at main office 


90 


95 


-5- 


88 


97 


-9” 


Hall passes required to leave class 


82 


87 


-5" 


79 


88 


-9” 


(N) 


1.241-1.377 






1.241- 1.377 







Computer Source— MPR: %40I02.do, fey40I03d.do. 
See notes to Table 6. 
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TABLE 1 1 



STUDENT PERCEPTIONS OF SCHOOL CLIMATE 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 

(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5) 


Switch to 
Private School 
(6)' 


Student reports (percentages) 














Students are proud to attend this 
school 


61 


56 


5* 


64 


56 


8* 


Behavior rules are strict 


67 


62 


5* 


69 


60 


9* 


Students get along with teachers 


60 


51 


9'" ■ 


65 


49 


16”' 


Feel “put down” by teachers 


17 


21 




15 


22 


-7' 


Teachers ignore cheating 


15 


18 


-3 


13 


17 


-4 


There is a lot of cheating in this 
school 


25 


30 


-5” 


18 


28 


-10” 


(N) 


1,618-1,746 






1,607-1,735 







Computer Source — MPR: sty40l 12.do, sty40I l3.do, sty40l l4.do, sty40l l5.do. 
See notes to Table 6. 
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TABLE 12 



HOMEWORK 

(Percentages) 





Scholarship 
Offered 
(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Compiler 

(5) 


Switch to 
Private School 
(6) 


Parent reports 














Child has more than one hour of 
homework 


57 


42 


15*” 


63 


37 


26*” 


Homework too easy 


8 


18 


-10*" 


6 


23 


-17*” 


(N) 


1.361-1.395 






1,361-1395 






Student reports 














Trouble keeping up with 
homework 


24 


24 


0 


24 


24 


0 


Time spent on homework on 
typical night (in minutes) 


47 


44 


3 


50 


45 


5 


Teachers return homework always 
or most of time 


52 


55 


-3 


51 


56 


-5 


(N) 


1.736-1.779 






1,726-1.751 







Computer Source —MPR: fay40l03c.do, fay40l03d.do, sty40l 13.do, sty40l l5.do. 



See notes to Table 6. 





TABLE 13 



SCHOOL COMMUNICATION WITH PARENTS 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5) 


Switch to 
Private School 
(6) 


Parents regularly informed about 
student grades 


91 


83 




97 


83 


14- 


Parents receive notes from teacher 


90 


80 


10‘" 


93 


76 


17- 


Parents receive newsletter 


82 


72 


10‘" 


84 


68 


16‘” 


Notified of disruptive behavior 


91 


81 


10‘" 


91 


75 


16‘" 


Parents speak to classes about jobs 


44 


37 


7" 


38 


26 


12" 


Parents participate in instruction 


65 


53 


12‘" 


67 


47 


20‘" 


Parent night 


92 


87 




97 


89 


8‘" 


Regular parent-teacher conferences 


93 


92 


1 


95 


93 


2 


(N) 


1,091-1,361 






1,091-1,361 







Computer Source — MPR: fay40I03c.do. 



See notes to Table 6. 
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TABLE 14 



RELIGIOUS PRACTICES 
(Percentages) 





Scholarship 
Offered 
(1) 


Control 
Group 
‘ (2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5) 


Switch to 
Private School 
(6) 


Student reports 














Religious instruction outside school 


20 


19 


1 


21 


19 


2 


Attend religious services 


52 


38 


14*** 


62 


37 


25*** 


Participate in church group 


44 


34 


10*** 


47 


28 


19*** 


(N) 


1,732-1,747 






1,721-1,736 







Computer Source— MPR: sty40I I2.do, sty40I I3.do. 
See notes to Table 6. 
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TABLE 15 



PARENTAL INVOLVEMENT IN CHILD’S EDUCATION 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

C21 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5) 


Switch to Private 
School 
(6) 


Average number of times per 
month parents did the following:^ 

Helped child with homework 


10 


11 


.1- 


9 


11 


-2"* 


Helped child with reading, 
mathematics 


8 


9 


-1" 


8 


9 


-1" 


Talked with child about school 


. 13 


14 


-1" 


13 


14 


-1" 


Attend school activity with child 


4 


5 


-1" 


4 


5 


-I" 


Worked on school projects 


5 


5 


0 


5 


6 


-1 


(N) 


1,377-1,391 






1,377-1,391 







Computer Source — MPR: fay40I07.do. 



See notes to Table 6. 

^The index is scored 0 if a parent never did the activity, 3 for 1-5 times, 8 for 6-10 times, 13 for 11-15 times, and 18 for 16 or 
more times in the past month. 
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TABLE 16 



STUDENT FRIENDSHIPS, BEHAVIOR, AND EXPECTATIONS 
(Percentages) 



Scholarship 




Scholarship 


Scholarship 


Control- 

Group 


Switch to Private 


Offered 


Control Group 


Offer Impact 


User 


Complier 


School 


LD 


(2) 


(21 


w 


(5) 


(6) 



Student reports 



Number of close fnends at school“ 


6 


6 


0 


6 


6 


0 


Percentage of students who feel 
“made fun of* by other students 


29 


38 


-9 • 


21 


35 


-15- 


Percentage of students who report 
that they will 














Graduate from high school 


66 


63 


3 


71 


66 


5 


Go on for further education after 
they leave high school 


73 


71 


2 


71 


68 


3 


Number of close friends who 












- 


Get in trouble with teachers 


2.0 


2.1 


-0.1 


1.9 


2.0 


-0.1 


Use bad language 


1.8 


2.2 


-0.4** 


1.7 


2.3 


-0.6 


Smoke cigarettes 


0.2 


0.2 


-0.1 


0.1 


0.2 


-0.1 


Drink beer or alcohol 


0.2 


0.2 


0.0 


0.2 


0.2 


0.0 


Use illegal drugs 


0.2 


0.2 


-0.1 


0.1 


0.3 


-0.1 


Like school 


4.0 


4.0 


0.0 


3.9 


3.8 


0.1 


Get good grades 


4.5 


4.3 


0.2* 


4.6 


4.3 


0.3* 


Parent reports (percentage) 














Child suspended in past year 


4 


7 


-3* 


2 


7 


-5* 


(N) 


1,305-1,730 






1,305-1,719 







Computer Source— MPR: fay40I02.do, sty40I I2.do, sty40I I3.do. 
See notes to Table 6. 



“The index is scored 0 if the child reports no close fnends at school, 1.5 for one to two fnends, 3.5 for three to four fnends, 5.5 
for five to six fnends, and 7.5 for seven or nK>re fnends. The same scoring is used for the number of fnends who get in trouble at 
school, who smoke cigarettes, who drink beer or alcohol, and who use illegal drugs. 
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TABLE 17 



PARENTAL AND STUDENT SATISFACTION WITH SCHOOL 
(Percentage) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer Impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5J 


Switch to Private 
School 
(6) 


Parent reports 














(Percent very satisfied) 














Observe religious traditions 


32 


30 


2- 


41 


6 


35- 


Class size 


25 


12 


13- 


30 


8 


22- 


Discipline 


38 


17 


21- 


51 


16 


35’" 


Academic quality 


37 


17 


20"’ 


48 


14 


34- 


Student respect for teachers 


39 


17 


22- 


50 


12 


38- 


Parental support 


26 


15 


11 - 


33 


14 


19- 


Teaching values 


34 


17 


17- 


44 


15 


29- 


What taught in school 


35 


17 


18’" 


45 


14 


31 - 


School safety 


40 


21 


19- 


48 


15 


33’" 


Teaching 


41 


20 


21- 


52 


16 


36’" 


Teacher-parent communication 


41 


25 


16- 


51 


24 


27- 


Clarity of school goals 


30 


16 


14- 


38 


14 


24"’ 


Staff teamwork 


25 


13 


12- 


31 


10 


21- 


Sports program 


16 


9 


?... 


20 


8 


12- 


School facility 


24 


12 


12’" 


29 


9 


20’” 


Parental involvement 


28 


17 


11 - 


34 


16 


18’“ 


Location 


47 


27 


20”’ 


55 


21 


34’" 


Gave school an “A” 


32 


13 


19— 


42 


10 


32- 


(N) 


1,366-1,398 






1,366-1,398 






Student reports 














Gave school an “A” 


43 


39 


4 


44 


36 


8 


Gave school “D”,“F’ 


6 


9 


-3‘ 


4 


9 


-5’ 


(N) 


1,805 






1,794 







Computer Source — MPR: fay40I02.do, sty40I I2.do, sty401 13.do. 
See notes to Table 6. 
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TABLE 18 



STUDENTS CHANGING SCHOOL DURING SCHOOL YEAR 
(Percentages) 





Scholarship 

Offered 

in 


Control 
Group 
(2) 


Scholarship 
Offer Impact 

L3) 


Scholarship 

User 

(4) 


Control- 

Group 

Complier 

(5) 


Switch to Private 
School 
(6) 


Attended same school for entire 


school year 


95 


93 


2 


100 


97 


3 


Reasons why did not attend same 
school for entire year 


Moved away 


1 


2 


-1 


0 ’ 


2 


-2 


(Quality of school 


0 


0 


0 


-1 


0 


-1 


School too expensive 


1 


1 


0 


1 


0 


1 


Suspended/expel led 


0 


0 


0 


0 


-1 


1 


Preferred public school 


0 


0 


0 


0 


0 


0 


Inconvenient location 


0 


1 


-1 


0 


1 


-1 


Preferred private school 


0 


-1 


1 


0 


-1 


1 


(N) 


1,400 






1,400 







Computer Source — MPR: fay40I03c.do. 
See notes to Table 6. 
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TABLE 19 



SCHOOL MATRICULATION PLANS FOR NEXT SCHOOL YEAR 
(Percentages) 





Scholarship 

Offered 

(1) 


Control 

Group 

(2) 


Scholarship 
Offer impact 
(3) 


Scholarship 

User 

(4) 


Control- 

Group 

Compiler 

15) 


Switch to Private 
School 
(6) 


Child will attend same school 
next year 


87 


76 


11 ... 


79 


60 


19- 


Reasons why student will not 
attend same school next year 














Quality of school 


7 


8 


-1 


4 


6 


-2 


Moving 


7 


5 


2 


6 


2 


4 


Graduating 


8 


15 


.7- 


5 


16 


.11- 


Preferred private school 


3 


1 


2* 


2 


-1 


3* 


Inconvenient location 


3 


3 


0 


2 


1 


1 


School too expensive 


4 


1 


3** 


4 


0 


4- 


Children in same school 


2 


2 


0 


1 


0 


1 


Asked not to return 


0 


0 


0 


0 


1 


-1 


Preferred public school 


0 


0 


0 


0 


0 


0 


(N) 


1,400-1,401 






1,400-1,401 







Computer Source— MPR: fey40I03h.do. 
See notes to Table 6. 
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YEAR-ONE, -TWO, AND -THREE READING TEST-SCORE IMPACTS 
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TABLE A-1 



DIFFERENCES IN BASELINE CHARACTERISTICS FOR THIRD FOLLOW-UP 
TREATMENTS AND CONTROLS 
(MEAN VALUES REPORTED) 



Variable 


Treatments 


Controls 


Difference 


t-statistic 


Significance 


Grade of Student (1996-1997) 


Kindergarten 


0.17 


0.16 


0.01 


0.31 


- 


First 


0.20 


0.20 


0.01 


0.32 


- 


Second 


0.21 


0.22 


-0.02 


-0.70 


- 


Third 


0.20 


0.24 


-0.04 


-1.59 


- 


Fourth 


0.22 


0.17 


0.04 


1.91 




Years student attended this school 


2.36 


2.47 


-0.11 


-1.51 


- 


Satisfaction with aspects of current school 


Location 


3.03 


2.94 


0.08 


1.73 




School safety 


2.80 


2.76 


0.04 


0.90 


- 


Teaching 


2.66 


2.69 


-0.03 


-0.63 


- 


How much school involves parents 


2.73 


2.72 


0.01 


0.27 


- 


Class sizes 


2.34 


2.33 


0.00 


0.06 


- 


School facilities 


2.60 


2.64 


-0.05 


-0.99 


- 


Student respect of teachers 


2.89 


2.86 


0.02 


0.51 


- 


Parent-teacher communication 


2.81 


2.82 


0.00 


-0.04 


- 


Observance of religious traditions 


2.24 


2.33 


-0.09 


-1.71 




Student in gifted classes 


0.11 


0.13 


-0.02 


-1.27 


- 


Student received help for disability 


0.12 


0.10 


0.02 


1.15 


- 


Mother's educational expectations for child 


16.73 


16.69 


0.04 


0.43 


- 



(10=some high school, 12=high school 
graduation, 14=some college, 16=college 
graduation, 18=more than college) 

Education level of mother or female guardian 



Some high school (did not graduate) 


0.21 


0.21 


0.00 


0.06 


- 


High school graduate or GED certificate 


0.25 


0.28 


-0.03 


-1.06 


- 


Some college 


0.41 


0.40 


0.01 


0.40 


- 


Graduated from four-year college 


0.08 


0.07 


0.01 


0.73 


- 


More than four-year college degree 


0.02 


0.03 


-0.01 


-0.61 


- 


Don't know 


0.02 


0.01 


0.01 


1.32 


- 
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TABLE A- 1 (continued) 



Variable 

Race/ethnicity of mother/female guardian 



Treatments Controls Difference t-statistic Significance 



White 


0.05 


0.04 


0.01 


0.92 




Black 


0.48 


0.44 


0.03 


1.19 




Puerto Rican 


0.17 


0.20 


-0.03 


-1.40 




Hispanic other than Puerto Rican 


0.25 


0.28 


-0.03 


-1.13 




Other 


0.05 


0.04 


0.01 


1.26 


- 


Birth place of mother/female guardian 


Bom in United States 


0.60 


0.62 


-0.02 


-0.59 




Bom in Puerto Rico 


0.07 


0.08 


-0.01 


-0.77 




Bom outside U.S. and Puerto Rico 


0.32 


0.29 


0.03 


1.08 


- 


Length of residence of mother in months 


36.07 


36.50 


-0.44 


-0.65 


- 


Job status of mother/female guardian 


Full-time job 


0.23 


0.20 


0.03 


1.15 




Part-time job 


0.15 


0.15 


0.00 


0.23 


_ 


Not working now but looking for work 


0.46 


0.46 


0.00 


-0.07 




Not working and not looking for work 


0.14 


0.16 


-0.02 


-0.92 




Don't know 


0.01 


0.02 


-0.01 


-1.42 


- 


Religious affiliation of female guardian 


Catholic 


0.50 


0.54 


-0.04 


-1.39 




Religion other than Catholic 


0.45 


0.41 


0.04 


1.41 




None 


0.05 


0.05 


0.00 


-0.02 


- 


Number of children in home 


2.41 


2.44 


-0.02 


-0.30 


- 


In child's home (percent saying yes) 


A daily newspaper 


0.85 


0.84 


0.01 


0.67 




An encyclopedia 


0.70 


0.70 


0.00 


-0.13 




A dictionary 


0.97 


0.98 


0.00 


-0.17 




More than 50 books 


0.86 


0.85 


0.01 


0.35 


- 


Member of household receiving assistance 


Food stamps 


0.66 


0.67 


-0.01 


-0.40 


_ 


Welfare 


0.56 


0.58 


-0.02 


-0.75 


_ 


Social Security 


0.11 


0.11 


0.00 


0.12 




Medicaid 


0.62 


0.67 


-0.05 


-1.76 




Supplemental Security Income 


0.14 


0.14 


-0.01 


-0.32 


- 


Family income 


9577.01 


9533.28 


43.74 


0.11 
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TABLE A-1 (continued) 



Variable 


Treatments 


Controls 


Difference 


t-statistic 


Significance 


Reading achievement scores 


Overall 


23.10 


25.36 


-2.26 


-1.59 


- 


First-grade cohort 


22.08 


29.01 


-6.93 


-2.15 


** 


Second-grade cohort 


26.54 


26.04 


0.50 


0.17 


- 


Third-grade cohort 


19.29 


22.63 


-3.35 


-1.47 


- 


Fourth-grade cohort 


24.68 


24.23 


0.45 


0.16 


- 


Mathematics achievement scores 


Overall 


17.85 


17.66 


0.19 


0.15 


- 


First-grade cohort 


9.78 


11.72 


-1.94 


-1.04 


- 


Second-grade cohort 


21.14 


19.42 


1.72 


0.73 


- 


Third-grade cohort 


16.09 


18.35 


-2.26 


-0.95 


- 


Four^-grade cohort 


24.46 


20.79 


3.68 


1.18 


- 


English spoken at home 


0.78 


0.75 


0.03 


1.35 


- 



COMPUTER Source — MPR: fay40101 .do, fa40101a.ado. 

^Significantly different from zero at the .10 level. 
••Significantly different from zero at the .05 level. 
•••Significantly different from zero at the .01 level. 
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TABLE A-2 



DIFFERENCES IN BASELINE CHARACTERISTICS FOR RESPONDENTS AND NONRESPONDENTS 
IN THIRD FOLLOW-UP; TREATMENT GROUP (MEAN VALUES REPORTED) 



Variable 


Respond- 

ents 


Non- 
respondents Difference 


t«statistic 


Significance 


Grade of Student (1996- 1997) 


Kindergarten 


0.16 


0.13 


0.03 


1.08 


_ 


First 


0.19 


0.18 


0.01 


0.24 


_ 


Second 


0.21 


0.21 


0.00 


0.15 


- 


Third 


0.22 


0.27 


-0.04 


-1.44 


_ 


Fourth 


0.21 


0.21 


0.00 


0.05 


- 


Years student attended this school 


2.43 


2.45 


-0.02 


-0.23 


- 


Satisfaction with aspects of current school 


Location 


3.04 


2.92 


0.12 


1.93 




School safety 


2.80 


2.67 


0.13 


2.08 




Teaching 


2.68 


2.54 


0.13 


2.15 




How much school involves parents 


2.72 


2.61 


0.11 


1.87 




Class sizes 


2.33 


2.33 


0.00 


-0.06 


_ 


School facilities 


2.61 


2.53 


0.09 


1.53 


_ 


Student respect of teachers 


2.89 


2.78 


0.12 


1.94 




Parent-teacher communication 


2.84 


2.71 


0.14 


2.28 




Observance of religious traditions 


2.26 


2.21 


0.06 


0.81 


- 


Student in gifted classes 


0.11 


0.13 


-0.02 


-0.84 


■- 


Student received help for disability 


0.11 


0.14 


-0.03 


-1.17 


- 


Mother's educational expectations for child 
(10=some high school, 12= high school 
graduation, 14=some college, 16=college 


graduation, 18=more than college) 


16.77 


16.58 


0.20 


1.52 


- 


Education level of mother or female guardian 


Some high school (did not graduate) 


0.23 


0.23 


0.01 


0.18 


_ 


High school graduate or GED certificate 


0.24 


0.25 


-0.01 


-0.21 


_ 


Some college 


0.41 


0.39 


0.01 


0.38 


_ 


Graduated from four-year college 


0.07 


0.08 


-0.01 


-0.45 


_ 


More than four-year college degree 


0.03 


0.03 


-0.01 


-0.54 


_ 


Don't know 


0.02 


0.01 


0.00 


0.32 




Race/ethnicity of mother/female guardian 


White 


0.03 


0.08 


-0.05 


-2.74 


««« 


Black 


0.45 


0.52 


-0.07 


-1.92 


« 


Puerto Rican 


0.18 


0.19 


-0.01 


-0.47 


_ 


Hispanic other than Puerto Rican 


0.29 


0.18 


0.11 


3.79 




Other 


0.05 


0.03 


0.02 


1.44 


- 



o 
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TABLE A-2 (Continued) 



Variable 


Respond- 

ents 


Non- 
respondents Difference 


t-statistic 


Significance 


Birth place of mother/female guardian 
Bom in United States 


0.57 


0.64 


-0.07 


-1.92 




Bom in Puerto Rico 


0.07 


0.09 


-0.02 


-0.86 


- 


Bom outside U.S. and Puerto Rico 


0.35 


0.27 


0.09 


2.54 




Length of residence of mother in months 


36.52 


35.11 


1.41 


1.57 


- 


Job status of moth er/fem ale guardian 


Full-time job 


0.22 


0.20 


0.02 


0.76 


- 


Part-time job 


0.16 


0.16 


-0.01 


-0.27 


- 


Not working now but looking for work 


0.45 


0.49 


. -0.03 


-0.87 


- 


Not working and not looking for work 


0.16 


0.14 


0.01 


0.49 


- 


Don’t know 


0.01 


0.01 


0.00 


0.53 


- 


Religious affiliation of female guardian 


Catholic 


0.53 


0.46 


0.07 


1.89 




Religion other than Catholic 


0.43 


0.48 


-0.06 


-1.60 


- 


None 


0.05 


0.06 


-0.01 


-0.66 


- 


Number of children in home 


2.45 


2.36 


0.09 


1.02 


- 


!n child's home (percent saying yes) 


A daily newspaper 


0.84 


0.86 


-0.02 


-0.70 


- 


An encyclopedia 


0.69 


0.72 


-0.02 


-0.66 


- 


A dictionary 


0.97 


0.97 


0.01 


0.50 


- 


More than 50 books 


0.84 


0.85 


-0.01 


-0.30 


- 



Member of household receiving assistance 
Food stamps 


0.65 


0.72 


-0.07 


-1.98 




Welfare 


0.54 


0.65 


-0.11 


-3.03 




Social Security 


0.12 


0.12 


0.00 


0.00 


- 


Medicaid 


0.63 


0.68 


-0.06 


-1.64 


- 


Supplemental Security Income 


0.14 


0.17 


-0.03 


-1.03 


- 


Family income 


9821.59 


8774.81 


1046.78 


2.17 
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TABLE A-2 (Continued) 



Variable 


Respond- 

ents 


Non- 
respondents Difference 


t-statistic 


Significance 


Reading achievement scores 


Overall 


22.74 


21.01 


1.73 


1.02 




First-grade cohort 


22.48 


23.62 


-1.14 


-0.27 




Second-grade cohort 


24.62 


20.17 


4.44 


1.12 




Third-grade cohort 


19.11 


20.21 


- 1.11 


-0.42 




Fourth-grade cohort 


25.17 


20.67 


4.49 


1.44 




Math achievement scores 


Overall 


17.66 


15.29 


2.37 


1.49 


_ 


First-grade cohort 


10.06 


9.22 


0.84 


0.31 


_ 


Second-grade cohort 


20.22 


18.62 


1.60 


0.54 




Third-grade cohort 


17.86 


13.47 


4.38 


1.64 


_ 


Fourth-grade cohort 


21.24 


19.44 


1.80 


0.45 


- 


English spoken at home 


0.74 


0.83 


-0.09 


-2.98 





Computer Source— MPR: fay40101 .do, fay40101 .ado. 



♦Significantly different from zero at the .10 level. 
♦♦Significantly different from zero at the .05 level. 
♦♦♦Significantly different from zero at the ,01 level. 
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TABLE A-3 



DIFFERENCES IN BASELINE CHARACTERISTICS FOR RESPONDENTS AND NONRESPONDENTS 



IN THIRD FOLLOW-UP: CONTROL GROUP (MEAN VALUES REPORTED) 



Non- 



Variable 


Respondents 


respondents 


Difference 


t-statistic Significance 


Grade of Student ( 1 996-1997) 


Kindergarten 


0.17 


0.14 


0.04 


1.41 


- 


First 


0.18 


0.15 


0.03 


1.17 


- 


Second 


0.22 


0.21 


0.01 


0.17 


. - 


Third 


0.24 


0.25 


-0.01 


-0.30 


- 


Fourth 


0.19 


0.25 


-0.06 


-2.03 




Years student attended this school 


2.45 


2.60 


-0.15 


-1.57 


- 


Satisfaction with aspects of current school 


Location 


2.92 


2.86 


0.06 


0.89 


- 


School safety 


2.73 


2.65 


0.08 


1.17 


- 


Teaching 


2.71 


2.57 


0.14 


2.17 




How much school involves parents 


2.74 


2.62 


0.12 


2.00 




Class sizes 


2.34 


2.23 


0.11 


1.66 




School facilities 


2.64 


2.51 


0.13 


2.01 




Student respect of teachers 


2.86 


2.74 


0.12 


1.77 




Parent-teacher communication 


2.82 


2.72 


0.10 


1.47 


- 


Observance of religious traditions 


2.31 


2.23 


0.08 


1.19 


- 


Student in gifted classes 


0.11 


0.12 


-0.01 


-0.26 


- 


Student received help for disability 


0.11 


0.11 


0.00 


-0.10 


- 


Mother's educational expectations for child 


16.64 


16.80 


-0.16 


-1.31 


_ 


(lO^some high school, 12=high school 
graduation, 14=some college, 16=college 
graduation, IS^more than college) 

Education level of mother or female 
guardian 


Some high school (did not graduate) 


0.24 


0.20 


0.03 


1.09 


- 


High school graduate or GED certificate 


0.28 


0.28 


0.00 


0.04 


- 


Some college 


0.39 


0.40 


-0.01 


-0.27 


- 


Graduated from four-year college 


0.06 


0.08 


-0.03 


-1.39 


- 


More than four-year college degree 


0.03 


0.02 


0.00 


0.25 




Don't know 


0.01 


0.01 


0.00 


-0.01 


- 


Race/ethnicity of mother/female guardian 


White 


0.03 


0.04 


-0.01 


-0.50 




Black 


0.42 


0.48 


-0.06 


-1.70 




Puerto Rican 


0.22 


0.22 


-0.01 


-0.18 


- ■ 


Hispanic other than Puerto Rican 


0.31 


0.23 


0.08 


2.52 




Other 


0.03 


0.04 


-0.01 


-0.38 


- 



o 
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TABLE A-3 (Continued) 



Non- 

Variable Respondents respondents Difference t-statistic Significance 

Birth place of mother/female guardian 



Bom in United States 


0.62 


0.61 


0.00 


0.08 




Bom in Puerto Rico 


0.09 


0.09 


0.01 


0.25 




Bom outside U.S. and Puerto Rico 


0.29 


0.30 


-0.01 


-0.28 


- 


Length of residence of mother in months 


36.97 


35.20 


1.77 


1.95 


* 


Job status of mother/female guardian 


Full-time job 


0.20 


0.25 


-0.05 


-1.53 




Part-time Job 


0.13 


0.18 


-0.05 


-1.67 


* 


Not working now but looking for work 


0.48 


0.41 


0.07 


1.99 




Not working and not looking for work 


0.16 


0.15 


0.01 


0.44 




Don’t know 


0.03 


0.02 


0.01 


0.95 


- 


Religious affiliation of female guardian 


Catholic 


0.57 


0.49 


0.08 


2.04 


** 


Religion other than Catholic 


0.39 


0.45 


-0.06 


-1.64 




None 


0.05 


0.06 


-0.02 


-0.88 


- 


Number of children in home 


2.48 


2.32 , 


0.16 


1.68 


* 


In child*s home (percent saying yes) 


A daily newspaper 


0.84 


0.86 


-0.02 


-0.76 




An encyclopedia 


0.70 


0.73 


-0.03 


-0.89 


. 


A dictionary 


0.98 


0.97 


0.01 


0.93 




More than 50 books 


0.86 


0.81 


0.05 


1.66 


* 


Member of household receiving assistance 


Food stands 


0.69 


0.66 


0.03 


0.77 


. 


Welfare 


0.61 


0.57 


0.04 


0.99 


_ 


Social Security 


0.13 


0.07 


0.06 


2.62 


*** 


Medicaid 


0.69 


0.62 


0.07 


1.89 


♦ 


Supplemental Security Income 


0.15 


0.12 


0.03 


1.10 


- 


Family income 


9,007.05 


10,097.17 


-1,090.11 


-1.97 




Reading achievement scores 


Overall 


23.26 


25.35 


-2.09 


-1.16 




First-grade cohort 


26.95 


31.27 


-4.32 


-1.03 


. 


Second-grade cohort 


23.92 


24.68 


-0.76 


-0.19 


_ 


Third-grade cohort 


20.01 


21.95 


-1.94 


-0.70 




Fourth-grade cohort 


23.07 


25.22 • 


-2.15 


-0.61 







TABLE A-3 (Continued) 



Variable 


Respondents 


Non- 

respondents 


Difference 


t-statistic Significance 


Math achievement scores 












Overall 


16.63 


18.84 


-2.21 


-1.29 


- 


First-grade cohort 


11.43 


13.88 


-2.45 


-0.88 




Second-grade cohort 


18.43 


19.91 


-1.49 


-0.43 


- 


Third-grade cohort 


15.58 


19.03 


-3.46 


-1.10 


- 


Fourth-grade cohort 


20.89 


21.24 


-0.35 


-0.09 


- 


English spoken at home 


0.73 


0.82 


-0.09 


-2.84 





Computer Source — MPR: fay40101.do, fay40101.ado. 

♦Significantly different from zero at the .10 level. 
♦♦Significantly different from zero at the .05 level. 
♦♦♦Significantly different from zero at the .01 level. 
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Appendix B 



Adjusting Sample Weights for Nonresponse 
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ADJUSTING SAMPLE WEIGHTS FOR NONRESPONSE 



Families within the sample had different probabilities of being offered a scholarship. To 
reflect these differences in the probability of selection and to reflect the composition of the 
population of eligible applicants, we weighted the sample data. The weights were constructed by 
taking the inverse of the probability of being selected for a scholarship. Weights for scholarship 
families were multiplied by .217 and weights for control-group families by .783 to reflect the 
ratio of treatment to control-group families in the initial pool of eligible applicants. All weights 
were divided by 2 to sum to the size of the population we were trying to represent, not twice the 
population. In this sample, the average wei^t was about 4.2. A family with a weight of 4.2 
stands in for 3.2 other families in the pool of applicants as well as itself. The weights, which 
were adjusted for the same family applying multiple times, range in size from about .5 to 22. 

About 18 percent of all families in the first year, 26 percent in the second year, and 28 
percent in the third year did not complete a survey. To adjust for such nonresponse, we 
computed the probability of responding based on a logit model.” The independent variables in 
the logit model included family characteristics such as race/ethnicity, number of siblings, 
language spoken at home, mother’s education, family income, and other variables used to stratify 
the sample when we collected the baseline data. After computing the predicted probability of 
responding, we adjusted the baseline weight as follows: . 

Wi=l/[fi*pi*pri], 

where fi includes the adjustment factors used for deriving the baseline weight^^,Pj is the 
probability of being selected for a scholarship (control group), pfjis the probability of 
responding to a follow-up survey, and Wiis the new weight variable. Families that did not 
respond to the follow-up survey were assigned a weight of zero. 

For the third-year student data, we foimd that 68 percent of the students responded' to the 
survey and that we had test scores for 67 percent. To adjust the weights for the student-level 
data, we followed the same procedures used for the parent data. 



Weights correspond only to the presence or absence of a family/student at a point in time 
and do not require that families or students are present in the sample for more than a single data 
session. 

The adjustment factors are as follows; (1) five discrete points at which families applied for 
scholarships; (2) whether a child attended a public school with below-average achievement; and 
(3) the munber of eligible children within the family (see Peterson, Myers, Haimson, and Howell 
1997). 
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Appendix C 



Analytic Approach 



ANALYTIC APPROACH 



For some analysts and program operators, the important policy question is as follows: what 
happens when a voucher or scholarship program is put into effect? But other analysts also want 
an answer to a second question: what is the impact of attending private school? Angrist, Imbens, 
and Rubin (1996) make some interesting distinctions between the two estimators used to address 
these two different questions. We refer to the first estimator as the intended to treat (ITT) 
estimator and the second as the complier average causal effect estimator (CAGE). The most 
important issues concerning the ITT estimator, which compares all children randomly assigned 
to the scholarship group with all children randomly assigned to the control group, are that among 
the children assigned to the two groups are ( 1 ) children who are induced by the offer of a 
scholarship to attend private school, ( 2 ) children who would have made the decision to attend 
private school regardless of the scholarship offer, and (3) children who would never attend 
private school. The CAGE estimator provides an estimate of the impact of the scholarship for 
only those who were or would have been induced by the offer of a scholarship to attend private 
school. Children who would have attended private school regardless of the offer of a scholarship 
and those who would have opted not to attend irrespective of the scholarship do not play a direct 
role in the estimated impact with the CAGE estimator. 

Computing Impacts of Being Offered a Scholarship (ITT) 

To compute the impact of being offered a scholarship, we use a simple statistical model that 
includes as independent variables an indicator for treatment status (offered a scholarship or in the 
control group) and a set of indicators that show the stratum from which a family was selected.^^ 
The strata are based on (1) five discrete points at which families applied for scholarships, (2) 
whether a child attended a public school with below-average achievement, and (3) the number of 
eligible children within the family. When computing the impact on student achievement test 
scores, we also included student baseline reading and mathematics achievement as independent 
variables. The basic form of the model is: 

= (G-1) 

where yj, is the outcome as measured in year three for respondent /; T, equals 1 if we offered a 
family a scholarship and 0 otherwise (families were randomly selected for the scholarship and 
control groups); Xi is a vector that includes indicator variables for each of the strata used in the 
random selection of scholarship families and baseline test scores when computing impacts on 
achievement; the random error term £ 3 , captures the effects of imobserved factors that influence 
the outcome; and the (3 s are parameters or vectors of parameters to be estimated. The parameter 
of most interest is because it shows the impact of being offered a scholarship on the outcome 



We estimate impacts for two points in time so we can assess change in impacts. 
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for year three. We estimate the model parameters by using ordinary least sqiiares for both 
categorical and continuous outcomes. Standard errors are computed by using the bootstrap. For 
a few binary outcomes, we checked the robustness of the ordinary least squares estimates by 
estimating a logit model. Similar qualitative conclusions held regardless of the method of 
estimation. 

Computing the Compiler Average Causal Effect (CACE) 

A simple comparison of an outcome for families in the scholarship group (those to whom we 
offered a scholarship) and the control group shows the impact of being offered a scholarship 
regardless of whether or not a family sent its child to private school. 

To compute the CACE estimator for the impact of ever attending private school, we need to 
estimate a statistical model that focuses on (1) the relationship between being offered a 
scholarship and ever attending private school and (2) the relationship between ever attending a 
private school and family and student outcomes. These relationships can be expressed as: 

Pj,=ao+a,7;+a2^,+e^3. 

where 7) equals 1 if we offered a family a scholarship and 0 otherwise (families were randomly 
selected for the scholarship and control groups); Xi is a vector that includes indicator variables 
for each of the strata used in the random selection of scholarship families and baseline test scores 
when computing impacts on achievement; equals 1 if ever attended private school and 0 
otherwise; ysi is the outcome of interest; and e^i, are random error terms that capture the 
effects of unobserved factors that influence both private-school attendance and the outcome; and 
a‘s and ffs are parameters or vectors of parameters to be estimated.^® The parameter of most 
interest is because it shows the impact of ever attending private school on the outcome.^’ 



For analyses of the parent and student survey data, we focused on attendance at private 
school in year three only. In this case. Pi, = 1 if attended private school in year three and 0 
otherwise. 

As already described in the report, we used two definitions of private-school attendance 
when anal>^ing the test scores: (1) ever attend private school and (2) attended private school for 
three years. The impacts for attending for t^ee years are implemented by making a small 
adjustment to the analytic models, which entails setting Py, = 1 if a student attended private 
school for three years andO otherwise. When using the instrumental variables estimator to 
estimate the impact of ever attending private school, we must assume that the scholarship offer 
had no impact on the year-three test scores for students who did not use a scholarship. To use 
the instrumental variables estimator to estimate the impact of attending private school for three 
years, we must make the same assumption as above and assume that the exposure to private 
schooling among students who attended for one or two years had no impact on the third-year test 
scores. 
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We estimate the model parameters by using the instrumental variables (IV) estimator. This 
technique allows us to compute asymptotically unbiased and efficient estimates of the 
parameters, which can be interpreted as the causal impact for compliers (students who were 
induced ever to attend private school by the scholarship offer) by using the fi'amework developed 
by Angrist, Imbens, and Rubin, 1996. To implement the instrumental variables estimator, we 
use the two-stage least squares procedure. 

In our tables we present (1) the impact of switching to private school, (2) the average of 
each outcome for families or students in the scholarship group who switched to private school 
(complied with the treatment protocol), and (3) the average of outcomes for families or students 
in the control group who would have attended private school if offered a scholarship. The first 
quantity is obtained from the statistical model described previously. The average for compliers 
in the control group is computed by adding the impact of attending private school to the average 
for members of the treatment group who would have complied. To compute the last quantity, we 
can use an alternative expression for computing the impacts of private-school attendance 
(compliance): 

E(/jCOMP = l)=[E(y\)-E(/jA = l)-Pr{A = 1)- 
E(y'^^ jN = \ )»Pr(N = 1) ]»Pr(COMP = 1)-' 

where 

E( y^ ^ / COMP = 1) = mean achievement at time = 3 for compliers (COMP = 1) 



E(y\) 
E(/JA = 1) 

E(/JN = 1) 

Pr (A=l) 

Pr (N=l) 

Pr (COMP=l) 



overall mean for the treatment group 

mean for students who would always attend private school 
regardless of the voucher offer 

mean for students who would never attend private school 
regardless of the voucher offer 

probability of always attending private school 

probability of never attending private school 

probability of complying 



The above expression tells us that the average of each outcome for compliers, which is 
unobserved, can be computed from known quantities from the treatment group and control 
group. 
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Model Specification for Looking at Grade-Specific Impacts and Between-Year Impacts on 
Reading and Mathematics Achievement 

Our analyses examined three hypotheses: 

• Average impacts on student and family outcomes were the same in years one, two, 
and three (and in year one and year two, and in year two and year three). 

• Grade-specific impacts on students’ reading and mathematics achievement test scores 
were similar within year one, year two, and year three. 

• Grade-specific impacts on students’ reading and mathematics achievement test scores 
were similar across years. 



To test these hypotheses, we constructed functions of the impact estimates and computed the 
standard errors of the fimctions by using the bootstrap method. Tables C-1 and C-2 list the 
specific functions. To implement our across-time and across- cohort analysis, we estimate 
equations similar to C. 1 and C.2 for earlier periods. 

For several reasons, we used the bootstrap to compute direct estimates of the standard errors. 
First, some analyses involve more than one child from each family, which produces clustering in 
the sample. To adjust for the clustering, we sampled families instead of children when 
constmcting the bootstrap samples. Second, the estimation of private-school impacts involved 
the use of the IV estimator and is complicated by the implicit presence of interaction terms in the 
model when comparing across-time or between-cohort impact estimates. To make these 
comparisons, we computed the functions in Tables C-1 and C-2 for each bootstrap sample and 
then computed the standard errors of the functions after 1,000 samples were formed and the 
models and fimctions were estimated. To assess whether overall impacts and grade-specific 
impacts changed between years, we computed some additional estimates, also indicated in Table 
C-2. 

To statistically test the hypotheses that involved making multiple comparisons, we used the 
Bonferonni procedure. The Bonferonni allows us to control the probability of making a type 1 
error when making multiple comparisons. To use the Bonferonni, we can take the probability of 
making a type 1 error for a z-test, for example, and divide it by the number of comparisons 
made. For example, if the probability is 0.10 and we are making four comparisons, then the 
critical value used for each comparison should be the value associated with a type 1 error of 
.025. By dividing by the number of planned comparisons, we implicitly set the probability of 
making one or more type 1 errors among the planned comparisons in the set to 0.10. This should 
be about equivalent to using an F-test in the usual setting when we want to test for differences 
between/among two or more means.^* 
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Similar procedures were used to assess change between years one and two and years two 
and three as well. 



TABLE C-1 



FUNCTIONS OF IMPACT ESTlMAl ES FUK UkAuE- 
SPECIFIC IMPACTS WITHIN YEARS" 
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Interpretation 



” ^ii 




Difference in 


1 

il 

< 


A, 


Difference in 


1 

il 

< 




Difference in 


> 

II 

”| 


/a, 


Difference in 


> 

II 

”| 


/41 


Difference in 


1 

il 

< 


/4> 


Difference in 


^12 ” ^12 


I22 


Difference in 


^22 ” A2” 


hi 


Difference in 


^32 ” ^12” 


hi 


Difference in 


^42 “ ^22” 


^2 


Difference in 


^52 “ ^22” 


hi 


Difference in 


^62 ” ^2” 


hi 


Difference in 


> 

II 


^23 


Difference in 


^23 ” ^13 


hi 


Difference in 


1 

II 

< 


hi 


Difference in 


1 

II 

< 


hi 


Difference in 


> 

II 

1 


hi 


Difference in i 


> 

II 

'"l 


hi 


Difference in i 



inq)acts for grades two and five — ^year one 
impacts for grades three and four — year one 
impacts for grades three and five — ^year one 
iirq)acts for grades four and five — year one 
impacts for grades three and four — year two 
impacts for grades three and five — year two 
impacts for grades three and six — ^year two 
impacts for grades four and five — year two 
impacts for grades four and six — year two 
impacts for grades five and six — year two 



“Grade-specific impacts refer only to analyses of achievement-test score impacts and not to analyses of 
family and student survey data. 
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TABLE C-2 



FUNCTIONS OF IMPACT ESTIMATES FOR 
BETWEEN- YEAR COMPARISONS 



Parameters of Interest 



Interpretation 



A, =/,, —7,2 
^2 “ ^21 ~ ^22 
^3 “^I “^2 

A4 = 74, ~ 1^2 

^5 ‘‘^Overall] Overall 2 
^6 “ I “ ^13 

^7“Ai“^23 
^8 “ “^3 
^9 “ Al “A3 

^10 ^^Overain ^^Overain 
^II “^12 “^13 

^12 “^22 “A3 

^13 “ A2 “A3 
^14 “ A 2 “A 3 

^15 “^^Overain ^^Overain 



Difference in inpacts for years one and two for grade three 
Difference in inpacts for years one and two for grade four 
Difference in inpacts for years one and two for grade five 
Difference in impacts for years one and two for grade six 
Difference in inpacts for years one and two overall 
Difference in impacts for years one and three for grade four 
Difference in impacts for years one and three for grade five 
Difference in impacts for years one and three for grade six 
Difference in inpacts for years one and three for grade seven 
Difference in impacts for years one and three overall 
Difference in inpacts for years two and three for grade four 
Difference in inpacts for years two and three for grade five 
Difference in inpacts for years two and three for grade six 
Difference in impacts for years two and three for grade seven 
Difference in inpacts for years two and three overall 



Note: 



Grade reported is the students’ grade at the latest year used in the difference calculation. For 
exanple, the difference in inpacts between years one and three for students in grade four refers 
to the cohort that was in grade two in year one and grade four in year three. 
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Test-Score Differences Between Cohorts 



o 

ERIC 



109 



YEAR ONE, TWO, AND THREE COMPOSITE TEST SCORE DIFFERENCES BETWEEN COHORTS 

(Percentile) 
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YEAR ONE, TWO, AND THREE READING TEST SCORE DIFFERENCES BETWEEN COHORTS 

(Percentile) 
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Appendix E 



African American and Latino Test-Score Differences 

Daniel P. Mayer 
David E. Myers 
Christina Tuttle 




-LID 



AFRICAN AMERICAN AND LATINO TEST-SCORE DIFFERENCES” 



Our analysis of the impact of educational vouchers on the test scores of African American 
and Latino students shows that vouchers had a positive impact only on test scores of African 
Americans. Further exploratory analyses reveal, however, that the African American students in 
private schools are not outperforming the Latino private-school students or even the Latino 
public-school students. A comparison of the average achievement test scores for African 
American and Latino students suggests that, after three years, the average reading and 
mathematics test scores of the Latinos who were offered a scholarship, of the Latino control 
students, and of the Afncan American students who were offered a scholarship were statistically 
similar while the average reading and mathematics test scores of the Afncan Americans in the 
control group were lower than the scores of students in any of the three other groups (see Tables 
20, 21, and 22). This pattern holds after statistically adjusting for baseline test scores, family 
income, education, and welfare status. Figure E-1 presents adjusted mean test scores for Latinos 
and Afncan Americans from base year through year three.^° 

The pattern is similar for the, reading, mathematics, and combined test scores. For instance, 
after one year, the adjusted mean for an Afncan American control is noticeably lower than the 
mean for a similar African American student who received a scholarship and a comparable 
Latino in the control group and for a Latino in the scholarship group. After three years, the 
adjusted means for the Latino voucher and control groups and for the African American 
scholarship group were similar. The adjusted mean for an African American control group 
student was 7.9 percentile points lower in reading, 8.3 points lower in mathematics, and 8.1 



Throughout this section, when necessary, we use regression analysis to impute values for 
school characteristic and family background to adjust for survey nonresponse, thereby 
maintaining identical samples as those presented in the previous section. Imputation was 
implemented at the family level separately by treatments and controls and by using race, baseline 
test scores, niunber of children in the family, lottery group, type of school at baseline, family 
income, and mother's education as predictors. For the indices, we included these backgroimd 
characteristics and nonmissing items from each index. For mother’s country of birth, if there 
was missing information in the baseline but answers on one of the follow-up surveys, we used 
the value from the follow-up survey. 

The models used to adjust the means included the baseline test score (except for the 
baseline test-score model), mother’s education level, family welfare status, family income, and 
the sample stratification variables. Adjusted means were calculated by setting these variables to 
the mean for the overall sample. These figures do not adjust for noncompliance (controls who 
went to private schools or treatments who did not use the scholarship), but the noncompliance 
rates of the two groups that took the examination in the third year are very similar: 86 percent of 
Afncan Americans and 85 percent of Latino controls went to public schools and 68 percent of 
Afncan Americans and 61 percent of Latinos ever switched to private schools, which suggests 
that test-score differences among groups were not a function of different compliance rates. 
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FIGURE E-1 

ADJUSTED TEST SCORE MEANS FOR AFRICAN AMERICANS AND LATINOS 
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performed significantly better in private schools than in public schools. Latino students 
performed as well as African American scholarship recipients regardless of which type of school 
they attended. 

This finding suggests that if African American and Latino students attended different public 
schools, there might be important differences in these schools that would explain why the 
African American control students performed relatively worse than the three other groups. We 
created an index of dissimilarity (a measure used to compare how mixed or segregated two 
groups are) to ascertain whether African Americans and Latinos in the control group attended 
different public schools. The measure ranges from 0 to 1 and shows the proportion of one group 
that would have to move in order to achieve the same distribution as the other group. We 
found that the students in our sample attended 420 public schools in the base year of Ae study 
and, by and large, that African American and Latino control-group students attended different 
schools. Seventy-seven percent of the African American students in our sample would have to 
move in order to achieve the same distribution among schools as the Latino students in the 
sample. This suggests that differences in the characteristics of public schools attended by 
African Americans and Latinos could explain why African Americans benefited from attending 
private schools, but Latinos did not. 

To assess whether school characteristics differed for African Americans and Latinos, we 
used two data sources: administrative data from the New York City public schools and parent 
survey data collected by MPR. The administrative data, which are from the base year (before 
random assignment),^^ include 10 indicators of school quality: (1) total school enrollment, (2) 
percentage of teachers at the school who are fully licensed and permanently assigned, (3) 
percentage of students who did not stay enrolled for the entire year, (4) percentage of days 
students were absent, (5) percentage of students suspended, (6) percentage of students involved 
in disciplinary incidents, (7) percentage of students receiving free limch, (8) percentage of 
students scoring at or above the state minimum the on the grade-three mathematics test, (9) 
percentage of students scoring at or above the state minimum on the grade-three reading test, and 
(10) pupil-teacher ratio. 

A comparison of characteristics of schools attended by African American and Latino 
students in our sample shows that the schools they attended in the baseline year did not differ 
along 6 of these 10 dimensions: total school enrollment, percentage of teachers at school who are 
fully licensed and permanently assigned, percentage of students who did not stay enrolled for the 



The index of dissimilarity is calculated in the following way: .05 * Z |Pii/PL-PiAA/PAA|: 
where Pjl is the population of Latinos in our study in base-year school i, and Pl is the total 
population of Latinos in our study. Piaa is the population of African Americans in our study in 
base-year school i, and Paa is the total population of African Americans on our study. 

Because only 50 percent of the control group reported that its children were in the same 
school from the base year through the third year of the study, these findings are only suggestive 
of the types of schools the African American and Latino students in our sample attended 
throughout the three years. 
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entire year, percentage of students suspended, percentage of students involved in disciplinary 
incidents, and percentage of students scoring at or above the state minimum on the grade-three 
mathematics test (Table 28).^^ 

The four dimensions on which the schools differed do not suggest that the schools attended 
by African Americans were worse than the schools attended by the Latinos. While absenteeism 
was higher in the schools attended by African American control-group students (1 1.5 percent of 
days versus 10.8 percent of days), the school poverty rates, measured by the percentage of 
students receiving a free lunch, and the pupil-teacher ratios were higher in the schools attended 
by Latinos in the control group (89.5 versus 86.6 percent receiving free lunch, and 1 7.2 versus 
16.3 pupil-teacher ratio). The percentage of students scoring at or above the state minimum on 
the grade-three reading test was higher in the schools attended by Latino schools (64.5 versus 
61.6 percent). 

While the administrative data reveal no clear pattern of differences, other important 
characteristics of schools related to quality, such as class size (rather than the broader measure of 
pupil-teacher ratio),^'’ the frequency of disruptions, the availability of various resources (such as 
a computer laboratory, library, or gymnasium), and the extent of communication between 
schools and parents, were not captured by these data. However, these characteristics are 
reflected in the data from our parent smvey. Specifically, parents provided information on the 
following school characteristics: 



• Disruptions: destruction of school property, tardiness, missing classes, fighting, 
cheating, racial conflict, carrying guns or other weapons, using drugs or alcohol 

• School resources: computer laboratory, library, gymnasium, cafeteria, child 
coimselors, nurse’s office 

• School programs: programs for non-English speakers, tutors for individual needs, 
programs for students with learning problems, programs for advanced learners, a 
music program, an arts program, an after-school program 

• School communication with parents: parents informed about student grades halfway 
through the grading periods, parents notified when students are sent to the office the 
first time for disruptive behavior, parents are asked to speak to classes about their 
jobs, parents participate in instruction, parent open-house or back-to-school night held 
at school, regular parent-teacher conferences, parents receive notes from teachers 



For these comparisons, school-level data were assigned to the students and then the 
difference in means was tested. 

Pupil-teacher ratios are an imprecise indicator of class size because they do not account 
for frequently intense teaching resources targeted toward Title I, special education, or bilingual 
services in some schools. When schools use those resources to attain low ratios for the targeted 
populations, they mask much higher ratios for the remaining classes in the school. 
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about student performance or behavior, parents receive a newsletter covering school 
activities/events 

• School size 

• Class size 

• Racial segregation in the classrooms^^ 



The parent survey data support the findings based on the administrative data (see Table 
E-1): along several dimensions, there were no differences in the public schools attended by 
African American and Latino students in our sample. Specific to the parent survey data, there 
were no differences in resources, class size, the frequency of school disruptions, the number and 
types of programs offered in the school, or the percentage of students attending completely 
segregated classrooms. The only significant difference was that, based on a comparison of 
parents’ reports, schools attended by African American students communicated less with parents 
than did schools attended by Latino students. 

Differences in school characteristics do not appear to explain the differences in test scores 
between the African American and Latino controls. And yet our examination of test-score trends 
over three years suggests that there was an interaction between the African American students 
and their public schools that may have affected them adversely when compared with the Latino 
students who remained in the public schools, the Latino students who attended private schools, 
and the African Americans who attended private schools. Knowing more about what makes the 
public-school experience for these two groups differ so much could provide important 
information regarding how to improve the public-school experience of African Americans. For 
the time being, however, the results presented here suggest that African Americans test scores 



Indices were created for four categories of school characteristics. Disruption is a 
cumulative measure of whether the parent indicated that the following were a somewhat serious 
or very serious problem at his or her child's school, with a maximum value of eight: students 
destroying property; tardiness; students missing classes; fighting; cheating; racial conflict; guns 
or other weapons; drugs or alcohol. Resources are measured on a scale of 0 to 6, with a point 
given for each of the following facilities that are available to students: computer laboratory; 
library; gynmasium; cafeteria; child counselors; and nurse’s office. Programs are similarly 
measured, on a scale of 0 to 7, with a point each for programs for non-English speakers, 
individual tutors, programs for students with learning problems, programs for advanced learners, 
a music program, an arts program, an after-school program. School communication is another 
cumulative measure; a point is given for whether the parent indicates that each of the following 
practices exists in his or her child's school (with a maximum of 8): parents are informed about 
student grades halfway through grading period; parents are notified when student sent to the 
office for the first time for disruptive behavior; parents speak to classes about their jobs; parents 
participate in instruction; a parent open-house or back-to-school night is held at school; parent- 
teacher conferences are held; parents receive notes about their student from the child's teachers; 
parents receive a newsletter about events in their child's school/classroom. 
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TABLE E-1 



AFRICAN AMERICAN AND LATINO CONTROL-GROUP SCHOOL CHARACTERISTICS 





African American 


Latinos 


Difference 
in Characteristics 


New York City Administrative Data 








Total school enrollment 


9.00 


9.47 


-0.47 


Percent of teachers at school who are fully 
licensed and permanently assigned 


77.75 


77.14 


0.61 


Percent of students who did not stay enrolled 
for the entire year 


90.86 


91.14 


-0.29 


Percent of days students were absent 


11.51 


10.64 


0.87*** 


Percent of students suspended 


1.99 


2.06 


-0.07 


Percent of students involved in disciplinary 
incidents 


1.18 


1.00 


0.18 


Percent of students receiving free lunch 


86.94 


89.50 


-2.56** 


Percent of students scoring at or above the state 
minimum the on grade-three mathematics 
test 


89.44 


90.20 


-0.75 


Percent of students scoring at or above the state 
minimum on grade-three reading test 


61.57 


64.45 


-2.88** 


Pupil-teacher ratio 


16.29 


17.38 


-1.09** 


Parent Survey Data 








Disruptions 


3.58 


3.77 


-0.19 


School resources 


5.52 


5.43 


0.09 


School programs 


5.43 


5.19 


0.23 


School communication 


5.29 


5.99 


-0.70*** 


Class size 


27.03 


27.35 


-0.32 


School size 


487.50 


509.38 


-21.88 


Racial segregation in the classrooms 


0.43 


0.39 


0.04 


(N) 


491-545 







Computer Source— MPR: fay40107j.do, fay40107k.do. 



'Impact is statistically significant at . 10 level, two-tailed test. 
"Impact is statistically significant at .05 level, two-tailed test. 
"'Impact is statistically significant at .01 level, two-tailed test. 
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will increase if African American students attend private schools, but the Latinos do as well in 
both public and private schools. 

Test-Score Impacts for First- and Later-Generation Latinos 

Research suggests that flrst-generation Latino students perform significantly better in the 
schools than their counterparts from later generations.^^ Given that there are a substantial munber 
of students from these two groups in our sample, we examined the impact on each group of the 
offer of a scholarship. In an exploratory analysis, we found no impact of the offer of a voucher 
based on the reading, mathematics, or combined test scores of later generations of Latino 
students, thereby suggesting that they perform similarly in public and private schools. For first- 
generation Latinos, although there is no impact on the combined test scores or on the 
mathematics scores, there is a negative impact on reading scores (see Table E-2).^’ 

First-generation Latinos who were offered a voucher scored 3.4 percentile points lower after 
three years in reading than the control-group students (effect size = .15). After controlling for 
family income, education, welfare status, and base-year test scores, we observed that the 
negative effect of a voucher was still present. Figure £-2 presents adjusted means for first and 
later generations of Latinos from the base year through year three. The adjusted means for 
reading in the third year show that a first-generation student who was offered a scholarship 
scored 3.2 percentile points lower than a similar student who was not offered a scholarship. No 
differences are apparent when examining the combined test scores or the mathematics test 
scores. 

Of particular interest with respect to the negative impact on the reading test scores of Latino 
students is the fact that first-generation Latinos who were offered a voucher were much less 
likely than the control group to receive English as a Second Language (ESL) services. While 79 
percent of Latino control-group students attend schools with special programs for non-English 
speakers, only 51 percent of treatment-group students attend schools with such programs. 
Including ESL services as a covariate in a regression analysis that tests treatment- and control- 
group differences in reading does not, however, reduce or eliminate the negative impact. 



A review of this literature is presented in Valenzuela (1999). 

We defined first- and later-generation status by sorting Latino students into two groups 
based on whether their mothers were or were not bom in the United States (first-generation 
students), as reported in the parent survey. 
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FIGURE E-2 

ADJUSTED TEST SCORE MEANS FOR FIRST AND LATER GENERATION LATINOS 
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Although we undertook this exploratory analysis when examining the difference in impacts for 
African American and Latino students, we want to emphasize that the negative impact on reading 
may be idiosyncratic and should be interpreted cautiously. 

There are two reasons why it may be idiosyncratic. First, at baseline, the reading test scores 
of the first-generation control students were statistically significantly higher than Aose of the 
first-generation treatment students, thereby suggesting that the randomization process did not 
create two equivalent groups in terms of reading scores.^* These differences were accounted for 
in the estimates presented for the follow-up years in Figure E-2 and in the negative impact 
reported in Table E-2, but there still may be non-observable characteristics contributing to the 
negative finding. Furthermore, a statistically significant impact is present only at the third 
follow-up and not at the first or second follow-up. 



Random assignment can break down when small samples of treatment- and control-group 
students are identified, and relatively large differences between the two groups for variables 
measured before randomization may be foimd. 
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